Devices, Methods, and Graphical User Interfaces for Interacting with Three-Dimensional Environments

ABSTRACT

A computer system displays a first computer-generated experience that includes displaying one or more virtual elements in a three-dimensional environment with a first appearance. While displaying the one or more virtual elements with the first appearance, the computer system receives biometric data corresponding to respiration of a first user. In response to receiving the biometric data: in accordance with a determination that the biometric data corresponding to the respiration of the first user meets first criteria, the computer system displays the one or more virtual elements in the three-dimensional environment with a second appearance that is different from the first appearance; and in accordance with a determination that the biometric data corresponding to the first user does not meet the first criteria, the computer system continues to display the one or more virtual elements in the three-dimensional environment with the first appearance.

RELATED APPLICATIONS

This application is a continuation application of U.S. application Ser.No. 17/483,730, filed Sep. 23, 2021, which claims priority to U.S.Provisional Patent Application 63/083,816, filed Sep. 25, 2020, whichare incorporated by reference in their entirety.

TECHNICAL FIELD

This disclosure relates generally to computer systems with a displaygeneration component and one or more input devices that provide computergenerated reality (CGR) experiences, including but not limited toelectronic devices that provide virtual reality and mixed realityexperiences via a display.

BACKGROUND

The development of computer systems for augmented reality has increasedsignificantly in recent years. Example augmented reality environmentsinclude at least some virtual elements that replace or augment thephysical world. Input devices, such as cameras, controllers, joysticks,touch-sensitive surfaces, and touch-screen displays for computer systemsand other electronic computing devices are used to interact withvirtual/augmented reality environments. Example virtual elements includevirtual objects include digital images, video, text, icons, and controlelements such as buttons and other graphics.

But methods and interfaces for interacting with environments thatinclude at least some virtual elements (e.g., applications, augmentedreality environments, mixed reality environments, and virtual realityenvironments) are cumbersome, inefficient, and limited. For example,systems that provide insufficient feedback for performing actionsassociated with virtual objects, systems that require a series of inputsto achieve a desired outcome in an augmented reality environment, andsystems in which manipulation of virtual objects are complex, tediousand error-prone, create a significant cognitive burden on a user, anddetract from the experience with the virtual/augmented realityenvironment. In addition, these methods take longer than necessary,thereby wasting energy. This latter consideration is particularlyimportant in battery-operated devices.

SUMMARY

Accordingly, there is a need for computer systems with improved methodsand interfaces for providing computer generated experiences to usersthat make interaction with the computer systems more efficient andintuitive for a user. The above deficiencies and other problemsassociated with user interfaces for computer systems with a displaygeneration component and one or more input devices are reduced oreliminated by the disclosed systems, methods, and user interfaces. Suchsystems, methods and interfaces optionally complement or replaceconventional systems, methods, and user interfaces for providingcomputer generated reality experiences to users. Such methods andinterfaces reduce the number, extent, and/or nature of the inputs from auser by helping the user to understand the connection between providedinputs and device responses to the inputs, thereby creating a moreefficient human-machine interface.

In accordance with some embodiments, a method is performed at a computersystem that is in communication with a first display generationcomponent and one or more first input devices. The method includes:displaying a first user interface object in a first view of athree-dimensional environment, wherein the three-dimensional environmentis at least partially shared between a first user and a second user,wherein the first user interface object is displayed with a first set ofappearance properties at a first position in the first view of thethree-dimensional environment; while displaying the first user interfaceobject with the first set of appearance properties at the first positionin the first view of the three-dimensional environment, detecting afirst user input provided by the first user, wherein the first userinput is directed to the first user interface object. The method furtherincludes: in response to detecting the first user input that is directedto the first user interface object: in accordance with a determinationthat the second user is not currently interacting with the first userinterface object, performing a first operation with respect to the firstuser interface object in accordance with the first user input; and inaccordance with a determination that the second user is currentlyinteracting with the first user interface object: displaying a visualindication that the first user interface object is not available forinteraction with the first user, wherein displaying the visualindication includes changing at least one of an appearance of the firstuser interface object or a position of the first user interface objectin the first view of the three-dimensional environment; and forgoingperforming the first operation with respect to the first user interfaceobject in accordance with the first user input.

In accordance with some embodiments, a method is performed at a computersystem that is in communication with a first display generationcomponent and one or more first input devices, including: while a firstuser is at a first location in a first physical environment, displayinga first view of a three-dimensional environment corresponding to a firstviewpoint that is associated with the first location in the firstphysical environment, wherein the first view of the three-dimensionalenvironment includes a first user interface object that represents afirst object in a second physical environment different from the firstphysical environment, wherein a respective position of the first userinterface object in the three-dimensional environment corresponds to arespective location of the first object in the second physicalenvironment in a first manner; detecting at least one of movement of thefirst user in the first physical environment and movement of the firstobject in the second physical environment; and in response to detectingthe at least one of movement of the first user in the first physicalenvironment and movement of the first object in the second physicalenvironment: displaying a second view of the three-dimensionalenvironment corresponding to a second viewpoint; and displaying thefirst user interface object in the second view of the three-dimensionalenvironment. Displaying the first user interface object in the secondview of the three-dimensional environment includes: in accordance with adetermination that the respective position of the first user interfaceobject in the three-dimensional environment that corresponds to therespective location of the first object in the second physicalenvironment in the first manner is more than a threshold distance from arespective position in the three-dimensional environment thatcorresponds to the second viewpoint associated with the second view ofthe three-dimensional environment, displaying the first user interfaceobject at a first display position in the second view of in thethree-dimensional environment, wherein the first display position is therespective position of the first user interface object in thethree-dimensional environment; and in accordance with a determinationthat the respective position of the first user interface object in thethree-dimensional environment that corresponds to the respectivelocation of the first object in the second physical environment in thefirst manner is less than the threshold distance from the respectiveposition in the three-dimensional environment that corresponds to thesecond viewpoint associated with the second view of thethree-dimensional environment, displaying the first user interfaceobject at a second display position in the second view of thethree-dimensional environment, wherein the second display position isoffset from the respective position of the first user interface objectin the three-dimensional environment.

In accordance with some embodiments, a method is performed at a computersystem that is in communication with a first display generationcomponent and one or more first input devices, including: displaying afirst computer-generated experience with a first level of immersion;while displaying the first computer-generated experience with the firstlevel of immersion, receiving biometric data corresponding to a firstuser; and in response to receiving the biometric data corresponding tothe first user: in accordance with a determination that the biometricdata corresponding to the first user meets first criteria, displayingthe first computer-generated experience with a second level ofimmersion, wherein the first computer-generated experience displayedwith the second level of immersion occupies a larger portion of a fieldof view of the first user than the first computer-generated experiencedisplayed with the first level of immersion; and in accordance with adetermination that the biometric data corresponding to the first userdoes not meet the first criteria, continuing to display the firstcomputer-generated experience with the first level of immersion.

In accordance with some embodiments, a method is performed at a computersystem that is in communication with a first display generationcomponent and one or more first input devices, including: displaying afirst view of a physical environment, wherein the first view of thephysical environment includes a first representation of a first portionof the physical environment; while displaying the first view of thephysical environment, detecting a first user input that corresponds to arequest to activate a first type of computer-generated sensoryadjustment of two or more types of computer-generated sensoryadjustments; and in response to detecting the first user input,displaying a second view of the physical environment, wherein the secondview of the physical environment includes a second representation of thefirst portion of the physical environment, wherein the secondrepresentation of the first portion of the physical environment has afirst display property that is adjusted relative to the firstrepresentation of the first portion of the physical environment inaccordance with the first type of computer-generated sensory adjustment;while displaying the second view of the physical environment, detectinga second user input that corresponds to a request to activate a secondtype of computer-generated sensory adjustment of the two or more typesof computer-generated sensory adjustments, wherein the second type ofcomputer-generated sensory adjustment is different from the first typeof computer-generated sensory adjustment; and in response to detectingthe second user input, displaying a third view of the physicalenvironment, wherein the third view of the physical environment incudesa third representation of the first portion of the physical environment,wherein the third representation of the first portion of the physicalenvironment has the first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with the first type of computer-generatedsensory adjustment, and a second display property that is adjustedrelative to the second representation of the physical environment inaccordance with the second type of computer-generated sensoryadjustment.

In accordance with some embodiments, a method is performed at a computersystem that is in communication with a first display generationcomponent and one or more first input devices, including: displaying afirst view of a three-dimensional environment, wherein the first view ofthe three-dimensional environment includes a first representation of afirst portion of a physical environment; while displaying the first viewof the three-dimensional environment including the first representationof the first portion of the physical environment, detecting movement ofa first user from a first location to a second location of the physicalenvironment; and in response to detecting the movement of the first userfrom the first location to the second location: in accordance with adetermination that the movement to the second location meets firstcriteria, wherein the first criteria include a first requirement thatthe second location corresponds to a location associated with a firsttype of exercise in order for the first criteria to be met, displaying asecond view of the three-dimensional environment, wherein the secondview of the three-dimensional environment includes a first set ofvirtual content corresponding to the first type of exercise, wherein thefirst set of virtual content replaces at least a portion of a secondrepresentation of a second portion of the physical environment thatincludes the second location; and in accordance with a determinationthat the movement to the second location meets second criteria,different from the first criteria, wherein the second criteria include asecond requirement that the second location corresponds to a locationassociated with a second type of exercise in order for the secondcriteria to be met, wherein the second type of exercise is differentfrom the first type of exercise, displaying a third view of thethree-dimensional environment, wherein the third view of thethree-dimensional environment includes a second set of virtual contentcorresponding to the second type of exercise, wherein the second set ofvirtual content is different from the first set of virtual content, andwherein the second set of virtual content replaces at least a portion ofa third representation of a third portion of the physical environmentthat includes the second location.

In accordance with some embodiments, a computer system includes or is incommunication with a display generation component (e.g., a display, aprojector, a head-mounted display, etc.), one or more input devices(e.g., one or more cameras, a touch-sensitive surface, optionally one ormore sensors to detect intensities of contacts with the touch-sensitivesurface), optionally one or more tactile output generators, one or moreprocessors, and memory storing one or more programs; the one or moreprograms are configured to be executed by the one or more processors andthe one or more programs include instructions for performing or causingperformance of the operations of any of the methods described herein. Inaccordance with some embodiments, a non-transitory computer readablestorage medium has stored therein instructions, which, when executed bya computer system with a display generation component, one or more inputdevices (e.g., one or more cameras, a touch-sensitive surface,optionally one or more sensors to detect intensities of contacts withthe touch-sensitive surface), and optionally one or more tactile outputgenerators, cause the device to perform or cause performance of theoperations of any of the methods described herein. In accordance withsome embodiments, a graphical user interface on a computer system with adisplay generation component, one or more input devices (e.g., one ormore cameras, a touch-sensitive surface, optionally one or more sensorsto detect intensities of contacts with the touch-sensitive surface),optionally one or more tactile output generators, a memory, and one ormore processors to execute one or more programs stored in the memoryincludes one or more of the elements displayed in any of the methodsdescribed herein, which are updated in response to inputs, as describedin any of the methods described herein. In accordance with someembodiments, a computer system includes: a display generation component,one or more input devices (e.g., one or more cameras, a touch-sensitivesurface, optionally one or more sensors to detect intensities ofcontacts with the touch-sensitive surface), and optionally one or moretactile output generators; and means for performing or causingperformance of the operations of any of the methods described herein. Inaccordance with some embodiments, an information processing apparatus,for use in a computer system with a display generation component, one ormore input devices (e.g., one or more cameras, a touch-sensitivesurface, optionally one or more sensors to detect intensities ofcontacts with the touch-sensitive surface), and optionally one or moretactile output generators, includes means for performing or causingperformance of the operations of any of the methods described herein.

Thus, computer systems with display generation components are providedwith improved methods and interfaces for interacting with athree-dimensional environment and facilitating the user's user of thecomputer systems when interacting with the three-dimensionalenvironment, thereby increasing the effectiveness, efficiency, and usersafety and satisfaction with such computer systems. Such methods andinterfaces may complement or replace conventional methods forinteracting with a three-dimensional environment and facilitating theuser's use of the computer systems when interacting with thethree-dimensional environment.

Note that the various embodiments described above can be combined withany other embodiments described herein. The features and advantagesdescribed in the specification are not all inclusive and, in particular,many additional features and advantages will be apparent to one ofordinary skill in the art in view of the drawings, specification, andclaims. Moreover, it should be noted that the language used in thespecification has been principally selected for readability andinstructional purposes, and may not have been selected to delineate orcircumscribe the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1 is a block diagram illustrating an operating environment of acomputer system for providing CGR experiences in accordance with someembodiments.

FIG. 2 is a block diagram illustrating a controller of a computer systemthat is configured to manage and coordinate a CGR experience for theuser in accordance with some embodiments.

FIG. 3 is a block diagram illustrating a display generation component ofa computer system that is configured to provide a visual component ofthe CGR experience to the user in accordance with some embodiments.

FIG. 4 is a block diagram illustrating a hand tracking unit of acomputer system that is configured to capture gesture inputs of the userin accordance with some embodiments.

FIG. 5 is a block diagram illustrating an eye tracking unit of acomputer system that is configured to capture gaze inputs of the user inaccordance with some embodiments.

FIG. 6 is a flowchart illustrating a glint-assisted gaze trackingpipeline in accordance with some embodiments.

FIGS. 7A-7C are block diagrams illustrating interaction with a userinterface object in a computer-generated three-dimensional environmentthat is shared between two or more users, in accordance with someembodiments.

FIGS. 7D-7F are block diagrams illustrating a method of displaying arepresentation of a physical object relative to a viewpoint of acurrently displayed view of a three-dimensional environment in differentmanners, where the viewpoint moves in accordance with movement of theuser in a first physical environment, the representation of the physicalobject moves in accordance with movement of the physical object in asecond physical environment different from the first physicalenvironment, and where a change in the manner of displaying therepresentation is triggered in response to a spatial relationshipbetween the representation of the physical object and the viewpointmeeting preset criteria, in accordance with some embodiments.

FIGS. 7G-7J are block diagrams illustrating changing a level ofimmersion with which an environment of a computer-generated experienceis displayed in accordance with changing biometric data of a user thatis received by the computer system, in accordance with some embodiments.

FIGS. 7K-7M are block diagrams illustrating aggregating the effects ofmultiple types of the sensory adjustment provided by a computer systemwhen displaying a view of an environment that includes a representationof a physical environment, in accordance with some embodiments.

FIGS. 7N-7P are block diagrams illustrating selectively displayingvirtual content that corresponds to a respective type of exercise in aview of a three-dimensional environment in accordance with adetermination that the portion of the physical environment in the viewof the three-dimensional environment corresponds to the respective typeof exercise, in accordance with some embodiments.

FIG. 8 is a flowchart of a method of supporting interaction with a userinterface object in a computer-generated three-dimensional environmentthat is shared between two or more users, in accordance with someembodiments.

FIGS. 9A-9B are a flowchart of a method of displaying a representationof a physical object relative to a viewpoint of a currently displayedview of a three-dimensional environment in different manners, where theviewpoint moves in accordance with movement of the user in a firstphysical environment, the representation of the physical object moves inaccordance with movement of the physical object in a second physicalenvironment different from the first physical environment, and where achange in the manner of displaying the representation is triggered inresponse to a spatial relationship between the representation of thephysical object and the viewpoint meeting preset criteria, in accordancewith some embodiments.

FIG. 10 is a flowchart of a method of changing a level of immersion withwhich an environment of a computer-generated experience is displayed inaccordance with changing biometric data of a user that is received bythe computer system, in accordance with some embodiments.

FIG. 11 is a flowchart of a method of aggregating the effects ofmultiple types of the sensory adjustment provided by a computer systemwhen displaying a view of an environment that includes a representationof a physical environment, in accordance with some embodiments.

FIG. 12 is a flowchart of a method of selectively displaying virtualcontent that corresponds to a respective type of exercise in a view of athree-dimensional environment in accordance with a determination thatthe portion of the physical environment in the view of thethree-dimensional environment corresponds to the respective type ofexercise, in accordance with some embodiments.

DESCRIPTION OF EMBODIMENTS

The present disclosure relates to user interfaces for providing acomputer generated reality (CGR) experience to a user, in accordancewith some embodiments.

The systems, methods, and GUIs described herein improve user interfaceinteractions with virtual/augmented reality environments in multipleways.

In some embodiments, the computer system permits multiple users to havethe right to access a first user interface object displayed in athree-dimensional environment, but prevents a user from accessing thefirst user interface object while another user is interacting with thefirst user interface object. When displaying a view of thethree-dimensional environment including the first user interface objectvia a first display generation component used by a first user, thecomputer system detects a first user input that is directed to the firstuser interface object. In response to detecting the first user input,the computer system, depending whether or not the first user interfaceobject is available for interaction with the first user at the time,performs a first operation corresponding to the first user input withrespect to the first user interface object, or displays a visualindication that the first user interface object is not available forinteraction with the first user and forgoes performance of the firstoperation. The computer system provides the visual indication andforgoes performance of the first operation in accordance with adetermination that another user has control of the first user interfaceobject at the time (e.g., another user is interacting with the firstuser interface object, is interacting with the first user interfaceobject in a manner that excludes the first user's contemporaneousinteraction, and/or has a lock on the first user interface object forthe type of action that the first user is attempting to perform, etc.).In some embodiments, displaying the visual indication includes movingthe first user interface object in the view of the three-dimensionalenvironment shown to the first user to maintain a preset distancebetween the first user interface object and the approachingrepresentation of the hand of the first user. In some embodiments,displaying the visual indication includes changing the visual appearanceof the first user interface object in the view of the three-dimensionalenvironment shown to the first user. In some embodiments, when the firstuser interface object is released to the first user by the controllinguser (e.g., by a throw gesture, a toss gesture, etc.), the computersystem rotates the first user interface object such that the first userinterface object is displayed with a preset orientation relative to theviewpoint of the currently displayed view of the three-dimensionalenvironment shown to the first user. In some embodiments, the computersystem provides controlling access to the first user interface object bydisplaying a representation of the first user interface object at aposition at or near the representation of a portion of the first user(e.g., in the representation of the hand of the first user, within anarm's reach of the virtual position of the user's face, etc.).Displaying a visual indication indicating that the first user interfaceobject is not available for interaction with the first user in the viewof the three-dimensional environment displayed via the displaygeneration component used by the first user, in response to the firstuser's attempt to interact with the first user interface object,provides intuitive and timely feedback at the time of attemptedinteraction, and reduces unnecessary visual clutter in the view of thethree-dimensional environment. Also, the same visual indication does notneed to be displayed to other users that is sharing the environment withthe first user, which reduces user confusion and improves efficiency ofthe man-machine interface.

In some embodiments, the computer system displays a view of athree-dimensional environment that includes a representation of aphysical object (e.g., a second user, an animal, a moving drone, etc.)that is located in a different physical environment from the physicalenvironment of a first user (and a first display generation componentused by the first user to view the three-dimensional environment). Thecomputer system, optionally, moves the viewpoint corresponding to thecurrently displayed view of the three-dimensional environment inaccordance with the movement of the first user (and/or the first displaygeneration component) in their physical environment. The computer systemdetermines the position and movement path of the representation of thephysical object in the three-dimensional environment based on a locationand movement path of the physical object in its physical environment.The computer system utilizes a first type of correspondence (e.g.,mapping and conversion relationships; optionally, different mapping andconversion relationships for the viewpoint, the physical object, and thefirst user, etc.) between positions in the three-dimensional environmentand locations in a respective physical environment (e.g., the physicalenvironment of the first user and the first display generationcomponent, the physical environment of the physical object, etc.). Undersome conditions (e.g., due to movement of the first user, and/ormovement of the physical object, etc.), the position of therepresentation of the physical object would be within a thresholddistance (e.g., an arm's length, three feet, a user-specified distance,etc.) of the position of the viewpoint of the currently displayed viewof the three-dimensional environment shown via the first displaygeneration component, if the position(s) are determined using the firsttype of correspondence between positions in the three-dimensionalenvironment and locations in the physical environments. Under suchconditions, the computer system displays the representation of thephysical object at an adjusted position that is offset from the positiondetermined based on the first type of correspondence. In someembodiments, the adjusted position is determined based on a second typeof correspondence that is different from the first type ofcorrespondence and ensures that the adjusted position remains more thanthe threshold distance from the position of the viewpoint of thecurrently displayed view of the three-dimensional environment shown viathe first display generation component. The computer system continues touse the second type of correspondence to determine the adjusted positionof the representation of the physical object, until the unadjustedposition calculated based on the first type of correspondence is morethan the threshold distance away from the position of the viewpoint ofthe currently displayed view of the three-dimensional environment shownvia the first display generation component. By monitoring the relativedistance between the position of the representation of the physicalobject and the position of the viewpoint of the currently displayed viewof the three-dimensional environment shown via the first displaygeneration component, the computer can timely adjust the displayedposition of the representation of the physical object, such that visualcollision between the viewpoint and the representation of the physicalobject can be avoided. This improves the user's visual experience, andreduces user confusion and mistakes when the user interacts with thethree-dimensional environment.

In some embodiments, the computer system changes the level of immersionwith which a computer-generated experience (e.g., visual experience,audio-visual experience, virtual reality experience, augmented realityexperience, etc.) is presented to a user in accordance with biometricdata corresponding to the user. For example, when the user is adjustinghis/her physical and emotional states, e.g., proactively or under theinfluence of the computer-generated content, after thecomputer-generated experience is started, the computer system may detectchanges in the biometric data (e.g., heart rate, blood pressure,breathing rate, etc.) corresponding to the user. In accordance with thechanges in the biometric data relative to respective sets of presetcriteria associated with different levels of immersion, the computersystem increases or decreases the level of immersion with which thecomputer-generated experience is provided to the user (e.g., by changingthe visual prominence (e.g., including spatial extent, visual depth,color saturation, visual contrast, etc.) of virtual content relative tothe visual prominence of the representation of the physical environment(e.g., by enhancing complexity, spatial extent, and/or visualcharacteristics of the virtual content, and/or reducing the visualclarity, blur radius, opacity, color saturation, etc. of therepresentation of the physical environment, etc.). Adjusting the levelof immersion with which a computer-generated experience is provided to auser based on changes in the biometric data corresponding to the userhelps the computer system to provide a smoother transition between aless immersive experience and a more immersive experience that bettercorresponds to the perceptive state of the user for thecomputer-generated experience, thereby reducing user confusion andimproving the efficacy of the computer-generated experience.

In some embodiments, the computer system provides multiple types ofsensory adjustment functions that enhance the user's ability to perceivedifferent aspects of a physical environment that may not be easilyperceivable without the aid of special equipment or the computer system.Instead of allowing the user to only use a single type of sensoryadjustment function when viewing a portion of a physical environment ata time, the computer system aggregates the effects of two or more typesof sensory enhancement functions on a representation of the portion ofthe physical environment, such that features and characteristics presentin the portion of the physical environment that were previously hiddenin the view of the physical environment provided by the computer systemmay be revealed. Allowing the effects of multiple types of sensoryadjustment functions to be aggregated on the representation of the sameportion of the physical environment and presented in a view of athree-dimensional environment that includes the representation of theportion of the physical environment enables the user to better perceiveand understand the physical environment, and improves the usefulness ofthe computer-generated view of the physical environment.

In some embodiments, the computer system displays virtual content (e.g.,virtual scenery, visual and functional enhancements of the exerciseequipment, user interfaces, health and score boards, etc.) thatcorresponds to a respective type of exercise in accordance with adetermination that the physical location represented in a view of athree-dimensional environment is associated with the respective type ofexercise. For example, as the user and the display generation componentmove from location to location in the real world, the virtual contentshown in the view of the three-dimensional environment is adjusted tocorrespond to the type of exercise that is associated with the currentlocation of the user and the display generation component. In someembodiments, when a location is associated with multiple types ofexercise, the computer system selects a type of exercise from themultiple types of exercises that are associated with the location basedon other contextual information (e.g., movement of the user, engagementof the user with the objects at the location, etc.), and displays thevisual content corresponding to the selected type of exercise.Automatically selecting and/or changing the virtual content based on therespective type of exercise that is associated with the location of theuser and the display generation component reduces the number, extent,and/or nature of the inputs from a user to achieve a desired outcome(e.g., selecting the suitable virtual content for a type of exercise,starting particular modes of exercise, etc.), thereby creating a moreefficient human-machine interface.

FIGS. 1-6 provide a description of example computer systems forproviding CGR experiences to users. FIGS. 7A-7C are block diagramsillustrating interaction with a user interface object in acomputer-generated three-dimensional environment that is shared betweentwo or more users, in accordance with some embodiments. FIGS. 7D-7F areblock diagrams illustrating a method of displaying a representation of aphysical object relative to a viewpoint of a currently displayed view ofa three-dimensional environment in different manners, where theviewpoint moves in accordance with movement of the user in a firstphysical environment, the representation of the physical object moves inaccordance with movement of the physical object in a second physicalenvironment different from the first physical environment, and where achange in the manner of displaying the representation is triggered inresponse to a spatial relationship between the representation of thephysical object and the viewpoint meeting preset criteria, in accordancewith some embodiments. FIGS. 7G-7J are block diagrams illustratingchanging a level of immersion with which an environment of acomputer-generated experience is displayed in accordance with changingbiometric data of a user that is received by the computer system, inaccordance with some embodiments. FIGS. 7K-7M are block diagramsillustrating aggregating the effects of multiple types of the sensoryadjustment provided by a computer system when displaying a view of anenvironment that includes a representation of a physical environment, inaccordance with some embodiments. FIGS. 7N-7P are block diagramsillustrating selectively displaying virtual content that corresponds toa respective type of exercise in a view of a three-dimensionalenvironment in accordance with a determination that the portion of thephysical environment in the view of the three-dimensional environmentcorresponds to the respective type of exercise, in accordance with someembodiments. The user interfaces in FIGS. 7A-7P are used to illustratethe processes in FIGS. 8-12 , respectively.

In some embodiments, as shown in FIG. 1 , the CGR experience is providedto the user via an operating environment 100 that includes a computersystem 101. The computer system 101 includes a controller 110 (e.g.,processors of a portable electronic device or a remote server), adisplay generation component 120 (e.g., a head-mounted device (HMD), adisplay, a projector, a touch-screen, etc.), one or more input devices125 (e.g., an eye tracking device 130, a hand tracking device 140, otherinput devices 150), one or more output devices 155 (e.g., speakers 160,tactile output generators 170, and other output devices 180), one ormore sensors 190 (e.g., image sensors, light sensors, depth sensors,tactile sensors, orientation sensors, proximity sensors, temperaturesensors, location sensors, motion sensors, velocity sensors, etc.), andoptionally one or more peripheral devices 195 (e.g., home appliances,wearable devices, etc.). In some embodiments, one or more of the inputdevices 125, output devices 155, sensors 190, and peripheral devices 195are integrated with the display generation component 120 (e.g., in ahead-mounted device or a handheld device).

When describing a CGR experience, various terms are used todifferentially refer to several related but distinct environments thatthe user may sense and/or with which a user may interact (e.g., withinputs detected by a computer system 101 generating the CGR experiencethat cause the computer system generating the CGR experience to generateaudio, visual, and/or tactile feedback corresponding to various inputsprovided to the computer system 101). The following is a subset of theseterms:

Physical environment: A physical environment refers to a physical worldthat people can sense and/or interact with without aid of electronicsystems. Physical environments, such as a physical park, includephysical articles, such as physical trees, physical buildings, andphysical people. People can directly sense and/or interact with thephysical environment, such as through sight, touch, hearing, taste, andsmell.

Computer-generated reality: In contrast, a computer-generated reality(CGR) environment refers to a wholly or partially simulated environmentthat people sense and/or interact with via an electronic system. In CGR,a subset of a person's physical motions, or representations thereof, aretracked, and, in response, one or more characteristics of one or morevirtual objects simulated in the CGR environment are adjusted in amanner that comports with at least one law of physics. For example, aCGR system may detect a person's head turning and, in response, adjustgraphical content and an acoustic field presented to the person in amanner similar to how such views and sounds would change in a physicalenvironment. In some situations (e.g., for accessibility reasons),adjustments to characteristic(s) of virtual object(s) in a CGRenvironment may be made in response to representations of physicalmotions (e.g., vocal commands). A person may sense and/or interact witha CGR object using any one of their senses, including sight, sound,touch, taste, and smell. For example, a person may sense and/or interactwith audio objects that create 3D or spatial audio environment thatprovides the perception of point audio sources in 3D space. In anotherexample, audio objects may enable audio transparency, which selectivelyincorporates ambient sounds from the physical environment with orwithout computer-generated audio. In some CGR environments, a person maysense and/or interact only with audio objects.

Examples of CGR include virtual reality and mixed reality.

Virtual reality: A virtual reality (VR) environment refers to asimulated environment that is designed to be based entirely oncomputer-generated sensory inputs for one or more senses. A VRenvironment comprises a plurality of virtual objects with which a personmay sense and/or interact. For example, computer-generated imagery oftrees, buildings, and avatars representing people are examples ofvirtual objects. A person may sense and/or interact with virtual objectsin the VR environment through a simulation of the person's presencewithin the computer-generated environment, and/or through a simulationof a subset of the person's physical movements within thecomputer-generated environment.

Mixed reality: In contrast to a VR environment, which is designed to bebased entirely on computer-generated sensory inputs, a mixed reality(MR) environment refers to a simulated environment that is designed toincorporate sensory inputs from the physical environment, or arepresentation thereof, in addition to including computer-generatedsensory inputs (e.g., virtual objects). On a virtuality continuum, amixed reality environment is anywhere between, but not including, awholly physical environment at one end and virtual reality environmentat the other end. In some MR environments, computer-generated sensoryinputs may respond to changes in sensory inputs from the physicalenvironment. Also, some electronic systems for presenting an MRenvironment may track location and/or orientation with respect to thephysical environment to enable virtual objects to interact with realobjects (that is, physical articles from the physical environment orrepresentations thereof). For example, a system may account formovements so that a virtual tree appears stationery with respect to thephysical ground.

Examples of mixed realities include augmented reality and augmentedvirtuality.

Augmented reality: An augmented reality (AR) environment refers to asimulated environment in which one or more virtual objects aresuperimposed over a physical environment, or a representation thereof.For example, an electronic system for presenting an AR environment mayhave a transparent or translucent display through which a person maydirectly view the physical environment. The system may be configured topresent virtual objects on the transparent or translucent display, sothat a person, using the system, perceives the virtual objectssuperimposed over the physical environment. Alternatively, a system mayhave an opaque display and one or more imaging sensors that captureimages or video of the physical environment, which are representationsof the physical environment. The system composites the images or videowith virtual objects, and presents the composition on the opaquedisplay. A person, using the system, indirectly views the physicalenvironment by way of the images or video of the physical environment,and perceives the virtual objects superimposed over the physicalenvironment. As used herein, a video of the physical environment shownon an opaque display is called “pass-through video,” meaning a systemuses one or more image sensor(s) to capture images of the physicalenvironment, and uses those images in presenting the AR environment onthe opaque display. Further alternatively, a system may have aprojection system that projects virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface, sothat a person, using the system, perceives the virtual objectssuperimposed over the physical environment. An augmented realityenvironment also refers to a simulated environment in which arepresentation of a physical environment is transformed bycomputer-generated sensory information. For example, in providingpass-through video, a system may transform one or more sensor images toimpose a select perspective (e.g., viewpoint) different than theperspective captured by the imaging sensors. As another example, arepresentation of a physical environment may be transformed bygraphically modifying (e.g., enlarging) portions thereof, such that themodified portion may be representative but not photorealistic versionsof the originally captured images. As a further example, arepresentation of a physical environment may be transformed bygraphically eliminating or obfuscating portions thereof.

Augmented virtuality: An augmented virtuality (AV) environment refers toa simulated environment in which a virtual or computer generatedenvironment incorporates one or more sensory inputs from the physicalenvironment. The sensory inputs may be representations of one or morecharacteristics of the physical environment. For example, an AV park mayhave virtual trees and virtual buildings, but people with facesphotorealistically reproduced from images taken of physical people. Asanother example, a virtual object may adopt a shape or color of aphysical article imaged by one or more imaging sensors. As a furtherexample, a virtual object may adopt shadows consistent with the positionof the sun in the physical environment.

Hardware: There are many different types of electronic systems thatenable a person to sense and/or interact with various CGR environments.Examples include head mounted systems, projection-based systems,heads-up displays (HUDs), vehicle windshields having integrated displaycapability, windows having integrated display capability, displaysformed as lenses designed to be placed on a person's eyes (e.g., similarto contact lenses), headphones/earphones, speaker arrays, input systems(e.g., wearable or handheld controllers with or without hapticfeedback), smartphones, tablets, and desktop/laptop computers. A headmounted system may have one or more speaker(s) and an integrated opaquedisplay. Alternatively, a head mounted system may be configured toaccept an external opaque display (e.g., a smartphone). The head mountedsystem may incorporate one or more imaging sensors to capture images orvideo of the physical environment, and/or one or more microphones tocapture audio of the physical environment. Rather than an opaquedisplay, a head mounted system may have a transparent or translucentdisplay. The transparent or translucent display may have a mediumthrough which light representative of images is directed to a person'seyes. The display may utilize digital light projection, OLEDs, LEDs,uLEDs, liquid crystal on silicon, laser scanning light source, or anycombination of these technologies. The medium may be an opticalwaveguide, a hologram medium, an optical combiner, an optical reflector,or any combination thereof. In one embodiment, the transparent ortranslucent display may be configured to become opaque selectively.Projection-based systems may employ retinal projection technology thatprojects graphical images onto a person's retina. Projection systemsalso may be configured to project virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface. Insome embodiments, the controller 110 is configured to manage andcoordinate a CGR experience for the user. In some embodiments, thecontroller 110 includes a suitable combination of software, firmware,and/or hardware. The controller 110 is described in greater detail belowwith respect to FIG. 2 . In some embodiments, the controller 110 is acomputing device that is local or remote relative to the scene 105(e.g., a physical setting/environment). For example, the controller 110is a local server located within the scene 105. In another example, thecontroller 110 is a remote server located outside of the scene 105(e.g., a cloud server, central server, etc.). In some embodiments, thecontroller 110 is communicatively coupled with the display generationcomponent 120 (e.g., an HMD, a display, a projector, a touch-screen,etc.) via one or more wired or wireless communication channels 144(e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). Inanother example, the controller 110 is included within the enclosure(e.g., a physical housing) of the display generation component 120(e.g., an HMD, or a portable electronic device that includes a displayand one or more processors, etc.), one or more of the input devices 125,one or more of the output devices 155, one or more of the sensors 190,and/or one or more of the peripheral devices 195, or share the samephysical enclosure or support structure with one or more of the above.

In some embodiments, the display generation component 120 is configuredto provide the CGR experience (e.g., at least a visual component of theCGR experience) to the user. In some embodiments, the display generationcomponent 120 includes a suitable combination of software, firmware,and/or hardware. The display generation component 120 is described ingreater detail below with respect to FIG. 3 . In some embodiments, thefunctionalities of the controller 110 are provided by and/or combinedwith the display generation component 120.

According to some embodiments, the display generation component 120provides a CGR experience to the user while the user is virtually and/orphysically present within the scene 105.

In some embodiments, the display generation component is worn on a partof the user's body (e.g., on his/her head, on his/her hand, etc.). Assuch, the display generation component 120 includes one or more CGRdisplays provided to display the CGR content. For example, in variousembodiments, the display generation component 120 encloses thefield-of-view of the user. In some embodiments, the display generationcomponent 120 is a handheld device (such as a smartphone or tablet)configured to present CGR content, and the user holds the device with adisplay directed towards the field-of-view of the user and a cameradirected towards the scene 105. In some embodiments, the handheld deviceis optionally placed within an enclosure that is worn on the head of theuser. In some embodiments, the handheld device is optionally placed on asupport (e.g., a tripod) in front of the user. In some embodiments, thedisplay generation component 120 is a CGR chamber, enclosure, or roomconfigured to present CGR content in which the user does not wear orhold the display generation component 120. Many user interfacesdescribed with reference to one type of hardware for displaying CGRcontent (e.g., a handheld device or a device on a tripod) could beimplemented on another type of hardware for displaying CGR content(e.g., an HMD or other wearable computing device). For example, a userinterface showing interactions with CGR content triggered based oninteractions that happen in a space in front of a handheld or tripodmounted device could similarly be implemented with an HMD where theinteractions happen in a space in front of the HMD and the responses ofthe CGR content are displayed via the HMD. Similarly, a user interfaceshowing interactions with CGR content triggered based on movement of ahandheld or tripod mounted device relative to the physical environment(e.g., the scene 105 or a part of the user's body (e.g., the user'seye(s), head, or hand)) could similarly be implemented with an HMD wherethe movement is caused by movement of the HMD relative to the physicalenvironment (e.g., the scene 105 or a part of the user's body (e.g., theuser's eye(s), head, or hand)).

While pertinent features of the operation environment 100 are shown inFIG. 1 , those of ordinary skill in the art will appreciate from thepresent disclosure that various other features have not been illustratedfor the sake of brevity and so as not to obscure more pertinent aspectsof the example embodiments disclosed herein.

FIG. 2 is a block diagram of an example of the controller 110 inaccordance with some embodiments. While certain specific features areillustrated, those skilled in the art will appreciate from the presentdisclosure that various other features have not been illustrated for thesake of brevity, and so as not to obscure more pertinent aspects of theembodiments disclosed herein. To that end, as a non-limiting example, insome embodiments, the controller 110 includes one or more processingunits 202 (e.g., microprocessors, application-specificintegrated-circuits (ASICs), field-programmable gate arrays (FPGAs),graphics processing units (GPUs), central processing units (CPUs),processing cores, and/or the like), one or more input/output (I/O)devices 206, one or more communication interfaces 208 (e.g., universalserial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE802.16x, global system for mobile communications (GSM), code divisionmultiple access (CDMA), time division multiple access (TDMA), globalpositioning system (GPS), infrared (IR), BLUETOOTH, ZIGBEE, and/or thelike type interface), one or more programming (e.g., I/O) interfaces210, a memory 220, and one or more communication buses 204 forinterconnecting these and various other components.

In some embodiments, the one or more communication buses 204 includecircuitry that interconnects and controls communications between systemcomponents. In some embodiments, the one or more I/O devices 206 includeat least one of a keyboard, a mouse, a touchpad, a joystick, one or moremicrophones, one or more speakers, one or more image sensors, one ormore displays, and/or the like.

The memory 220 includes high-speed random-access memory, such as dynamicrandom-access memory (DRAM), static random-access memory (SRAM),double-data-rate random-access memory (DDR RAM), or other random-accesssolid-state memory devices. In some embodiments, the memory 220 includesnon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid-state storage devices. The memory 220 optionallyincludes one or more storage devices remotely located from the one ormore processing units 202. The memory 220 comprises a non-transitorycomputer readable storage medium. In some embodiments, the memory 220 orthe non-transitory computer readable storage medium of the memory 220stores the following programs, modules and data structures, or a subsetthereof including an optional operating system 230 and a CGR experiencemodule 240.

The operating system 230 includes instructions for handling variousbasic system services and for performing hardware dependent tasks. Insome embodiments, the CGR experience module 240 is configured to manageand coordinate one or more CGR experiences for one or more users (e.g.,a single CGR experience for one or more users, or multiple CGRexperiences for respective groups of one or more users). To that end, invarious embodiments, the CGR experience module 240 includes a dataobtaining unit 242, a tracking unit 244, a coordination unit 246, and adata transmitting unit 248.

In some embodiments, the data obtaining unit 242 is configured to obtaindata (e.g., presentation data, interaction data, sensor data, locationdata, etc.) from at least the display generation component 120 of FIG. 1, and optionally one or more of the input devices 125, output devices155, sensors 190, and/or peripheral devices 195. To that end, in variousembodiments, the data obtaining unit 242 includes instructions and/orlogic therefor, and heuristics and metadata therefor.

In some embodiments, the tracking unit 244 is configured to map thescene 105 and to track the position/location of at least the displaygeneration component 120 with respect to the scene 105 of FIG. 1 , andoptionally, to one or more of the input devices 125, output devices 155,sensors 190, and/or peripheral devices 195. To that end, in variousembodiments, the tracking unit 244 includes instructions and/or logictherefor, and heuristics and metadata therefor. In some embodiments, thetracking unit 244 includes hand tracking unit 243 and/or eye trackingunit 245. In some embodiments, the hand tracking unit 243 is configuredto track the position/location of one or more portions of the user'shands, and/or motions of one or more portions of the user's hands withrespect to the scene 105 of FIG. 1 , relative to the display generationcomponent 120, and/or relative to a coordinate system defined relativeto the user's hand. The hand tracking unit 243 is described in greaterdetail below with respect to FIG. 4 . In some embodiments, the eyetracking unit 245 is configured to track the position and movement ofthe user's gaze (or more broadly, the user's eyes, face, or head) withrespect to the scene 105 (e.g., with respect to the physical environmentand/or to the user (e.g., the user's hand)) or with respect to the CGRcontent displayed via the display generation component 120. The eyetracking unit 245 is described in greater detail below with respect toFIG. 5 .

In some embodiments, the coordination unit 246 is configured to manageand coordinate the CGR experience presented to the user by the displaygeneration component 120, and optionally, by one or more of the outputdevices 155 and/or peripheral devices 195. To that end, in variousembodiments, the coordination unit 246 includes instructions and/orlogic therefor, and heuristics and metadata therefor.

In some embodiments, the data transmitting unit 248 is configured totransmit data (e.g., presentation data, location data, etc.) to at leastthe display generation component 120, and optionally, to one or more ofthe input devices 125, output devices 155, sensors 190, and/orperipheral devices 195. To that end, in various embodiments, the datatransmitting unit 248 includes instructions and/or logic therefor, andheuristics and metadata therefor.

Although the data obtaining unit 242, the tracking unit 244 (e.g.,including the eye tracking unit 243 and the hand tracking unit 244), thecoordination unit 246, and the data transmitting unit 248 are shown asresiding on a single device (e.g., the controller 110), it should beunderstood that in other embodiments, any combination of the dataobtaining unit 242, the tracking unit 244 (e.g., including the eyetracking unit 243 and the hand tracking unit 244), the coordination unit246, and the data transmitting unit 248 may be located in separatecomputing devices.

Moreover, FIG. 2 is intended more as functional description of thevarious features that may be present in a particular implementation asopposed to a structural schematic of the embodiments described herein.As recognized by those of ordinary skill in the art, items shownseparately could be combined and some items could be separated. Forexample, some functional modules shown separately in FIG. 2 could beimplemented in a single module and the various functions of singlefunctional blocks could be implemented by one or more functional blocksin various embodiments. The actual number of modules and the division ofparticular functions and how features are allocated among them will varyfrom one implementation to another and, in some embodiments, depends inpart on the particular combination of hardware, software, and/orfirmware chosen for a particular implementation.

FIG. 3 is a block diagram of an example of the display generationcomponent 120 in accordance with some embodiments. While certainspecific features are illustrated, those skilled in the art willappreciate from the present disclosure that various other features havenot been illustrated for the sake of brevity, and so as not to obscuremore pertinent aspects of the embodiments disclosed herein. To that end,as a non-limiting example, in some embodiments the HMD 120 includes oneor more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs,CPUs, processing cores, and/or the like), one or more input/output (I/O)devices and sensors 306, one or more communication interfaces 308 (e.g.,USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x,GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and/or the like typeinterface), one or more programming (e.g., I/O) interfaces 310, one ormore CGR displays 312, one or more optional interior- and/orexterior-facing image sensors 314, a memory 320, and one or morecommunication buses 304 for interconnecting these and various othercomponents.

In some embodiments, the one or more communication buses 304 includecircuitry that interconnects and controls communications between systemcomponents. In some embodiments, the one or more I/O devices and sensors306 include at least one of an inertial measurement unit (IMU), anaccelerometer, a gyroscope, a thermometer, one or more physiologicalsensors (e.g., blood pressure monitor, heart rate monitor, blood oxygensensor, blood glucose sensor, etc.), one or more microphones, one ormore speakers, a haptics engine, one or more depth sensors (e.g., astructured light, a time-of-flight, or the like), and/or the like.

In some embodiments, the one or more CGR displays 312 are configured toprovide the CGR experience to the user. In some embodiments, the one ormore CGR displays 312 correspond to holographic, digital lightprocessing (DLP), liquid-crystal display (LCD), liquid-crystal onsilicon (LCoS), organic light-emitting field-effect transitory (OLET),organic light-emitting diode (OLED), surface-conduction electron-emitterdisplay (SED), field-emission display (FED), quantum-dot light-emittingdiode (QD-LED), micro-electro-mechanical system (MEMS), and/or the likedisplay types. In some embodiments, the one or more CGR displays 312correspond to diffractive, reflective, polarized, holographic, etc.waveguide displays. For example, the HMD 120 includes a single CGRdisplay. In another example, the HMD 120 includes a CGR display for eacheye of the user. In some embodiments, the one or more CGR displays 312are capable of presenting MR and VR content. In some embodiments, theone or more CGR displays 312 are capable of presenting MR or VR content.

In some embodiments, the one or more image sensors 314 are configured toobtain image data that corresponds to at least a portion of the face ofthe user that includes the eyes of the user (and may be referred to asan eye-tracking camera). In some embodiments, the one or more imagesensors 314 are configured to obtain image data that corresponds to atleast a portion of the user's hand(s) and optionally arm(s) of the user(and may be referred to as a hand-tracking camera). In some embodiments,the one or more image sensors 314 are configured to be forward-facing soas to obtain image data that corresponds to the scene as would be viewedby the user if the HMD 120 was not present (and may be referred to as ascene camera). The one or more optional image sensors 314 can includeone or more RGB cameras (e.g., with a complimentarymetal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device(CCD) image sensor), one or more infrared (IR) cameras, one or moreevent-based cameras, and/or the like.

The memory 320 includes high-speed random-access memory, such as DRAM,SRAM, DDR RAM, or other random-access solid-state memory devices. Insome embodiments, the memory 320 includes non-volatile memory, such asone or more magnetic disk storage devices, optical disk storage devices,flash memory devices, or other non-volatile solid-state storage devices.The memory 320 optionally includes one or more storage devices remotelylocated from the one or more processing units 302. The memory 320comprises a non-transitory computer readable storage medium. In someembodiments, the memory 320 or the non-transitory computer readablestorage medium of the memory 320 stores the following programs, modulesand data structures, or a subset thereof including an optional operatingsystem 330 and a CGR presentation module 340.

The operating system 330 includes instructions for handling variousbasic system services and for performing hardware dependent tasks. Insome embodiments, the CGR presentation module 340 is configured topresent CGR content to the user via the one or more CGR displays 312. Tothat end, in various embodiments, the CGR presentation module 340includes a data obtaining unit 342, a CGR presenting unit 344, a CGR mapgenerating unit 346, and a data transmitting unit 348.

In some embodiments, the data obtaining unit 342 is configured to obtaindata (e.g., presentation data, interaction data, sensor data, locationdata, etc.) from at least the controller 110 of FIG. 1 . To that end, invarious embodiments, the data obtaining unit 342 includes instructionsand/or logic therefor, and heuristics and metadata therefor.

In some embodiments, the CGR presenting unit 344 is configured topresent CGR content via the one or more CGR displays 312. To that end,in various embodiments, the CGR presenting unit 344 includesinstructions and/or logic therefor, and heuristics and metadatatherefor.

In some embodiments, the CGR map generating unit 346 is configured togenerate a CGR map (e.g., a 3D map of the mixed reality scene or a mapof the physical environment into which computer generated objects can beplaced to generate the computer generated reality) based on mediacontent data. To that end, in various embodiments, the CGR mapgenerating unit 346 includes instructions and/or logic therefor, andheuristics and metadata therefor.

In some embodiments, the data transmitting unit 348 is configured totransmit data (e.g., presentation data, location data, etc.) to at leastthe controller 110, and optionally one or more of the input devices 125,output devices 155, sensors 190, and/or peripheral devices 195. To thatend, in various embodiments, the data transmitting unit 348 includesinstructions and/or logic therefor, and heuristics and metadatatherefor.

Although the data obtaining unit 342, the CGR presenting unit 344, theCGR map generating unit 346, and the data transmitting unit 348 areshown as residing on a single device (e.g., the display generationcomponent 120 of FIG. 1 ), it should be understood that in otherembodiments, any combination of the data obtaining unit 342, the CGRpresenting unit 344, the CGR map generating unit 346, and the datatransmitting unit 348 may be located in separate computing devices.

Moreover, FIG. 3 is intended more as a functional description of thevarious features that could be present in a particular implementation asopposed to a structural schematic of the embodiments described herein.As recognized by those of ordinary skill in the art, items shownseparately could be combined and some items could be separated. Forexample, some functional modules shown separately in FIG. 3 could beimplemented in a single module and the various functions of singlefunctional blocks could be implemented by one or more functional blocksin various embodiments. The actual number of modules and the division ofparticular functions and how features are allocated among them will varyfrom one implementation to another and, in some embodiments, depends inpart on the particular combination of hardware, software, and/orfirmware chosen for a particular implementation.

FIG. 4 is a schematic, pictorial illustration of an example embodimentof the hand tracking device 140. In some embodiments, hand trackingdevice 140 (FIG. 1 ) is controlled by hand tracking unit 243 (FIG. 2 )to track the position/location of one or more portions of the user'shands, and/or motions of one or more portions of the user's hands withrespect to the scene 105 of FIG. 1 (e.g., with respect to a portion ofthe physical environment surrounding the user, with respect to thedisplay generation component 120, or with respect to a portion of theuser (e.g., the user's face, eyes, or head), and/or relative to acoordinate system defined relative to the user's hand. In someembodiments, the hand tracking device 140 is part of the displaygeneration component 120 (e.g., embedded in or attached to ahead-mounted device). In some embodiments, the hand tracking device 140is separate from the display generation component 120 (e.g., located inseparate housings or attached to separate physical support structures).

In some embodiments, the hand tracking device 140 includes image sensors404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and/orcolor cameras, etc.) that capture three-dimensional scene informationthat includes at least a hand 406 of a human user. The image sensors 404capture the hand images with sufficient resolution to enable the fingersand their respective positions to be distinguished. The image sensors404 typically capture images of other parts of the user's body, as well,or possibly all of the body, and may have either zoom capabilities or adedicated sensor with enhanced magnification to capture images of thehand with the desired resolution. In some embodiments, the image sensors404 also capture 2D color video images of the hand 406 and otherelements of the scene. In some embodiments, the image sensors 404 areused in conjunction with other image sensors to capture the physicalenvironment of the scene 105, or serve as the image sensors that capturethe physical environment of the scene 105. In some embodiments, theimage sensors 404 are positioned relative to the user or the user'senvironment in a way that a field of view of the image sensors or aportion thereof is used to define an interaction space in which handmovement captured by the image sensors are treated as inputs to thecontroller 110.

In some embodiments, the image sensors 404 outputs a sequence of framescontaining 3D map data (and possibly color image data, as well) to thecontroller 110, which extracts high-level information from the map data.This high-level information is typically provided via an ApplicationProgram Interface (API) to an application running on the controller,which drives the display generation component 120 accordingly. Forexample, the user may interact with software running on the controller110 by moving his hand 408 and changing his hand posture.

In some embodiments, the image sensors 404 project a pattern of spotsonto a scene containing the hand 406 and captures an image of theprojected pattern. In some embodiments, the controller 110 computes the3D coordinates of points in the scene (including points on the surfaceof the user's hand) by triangulation, based on transverse shifts of thespots in the pattern. This approach is advantageous in that it does notrequire the user to hold or wear any sort of beacon, sensor, or othermarker. It gives the depth coordinates of points in the scene relativeto a predetermined reference plane, at a certain distance from the imagesensors 404. In the present disclosure, the image sensors 404 areassumed to define an orthogonal set of x, y, z axes, so that depthcoordinates of points in the scene correspond to z components measuredby the image sensors. Alternatively, the hand tracking device 440 mayuse other methods of 3D mapping, such as stereoscopic imaging ortime-of-flight measurements, based on single or multiple cameras orother types of sensors.

In some embodiments, the hand tracking device 140 captures and processesa temporal sequence of depth maps containing the user's hand, while theuser moves his hand (e.g., whole hand or one or more fingers). Softwarerunning on a processor in the image sensors 404 and/or the controller110 processes the 3D map data to extract patch descriptors of the handin these depth maps. The software matches these descriptors to patchdescriptors stored in a database 408, based on a prior learning process,in order to estimate the pose of the hand in each frame. The posetypically includes 3D locations of the user's hand joints and fingertips.

The software may also analyze the trajectory of the hands and/or fingersover multiple frames in the sequence in order to identify gestures. Thepose estimation functions described herein may be interleaved withmotion tracking functions, so that patch-based pose estimation isperformed only once in every two (or more) frames, while tracking isused to find changes in the pose that occur over the remaining frames.The pose, motion and gesture information are provided via theabove-mentioned API to an application program running on the controller110. This program may, for example, move and modify images presented onthe display generation component 120, or perform other functions, inresponse to the pose and/or gesture information.

In some embodiments, the software may be downloaded to the controller110 in electronic form, over a network, for example, or it mayalternatively be provided on tangible, non-transitory media, such asoptical, magnetic, or electronic memory media. In some embodiments, thedatabase 408 is likewise stored in a memory associated with thecontroller 110. Alternatively or additionally, some or all of thedescribed functions of the computer may be implemented in dedicatedhardware, such as a custom or semi-custom integrated circuit or aprogrammable digital signal processor (DSP). Although the controller 110is shown in FIG. 4 , by way of example, as a separate unit from theimage sensors 440, some or all of the processing functions of thecontroller may be performed by a suitable microprocessor and software orby dedicated circuitry within the housing of the hand tracking device402 or otherwise associated with the image sensors 404. In someembodiments, at least some of these processing functions may be carriedout by a suitable processor that is integrated with the displaygeneration component 120 (e.g., in a television set, a handheld device,or head-mounted device, for example) or with any other suitablecomputerized device, such as a game console or media player. The sensingfunctions of image sensors 404 may likewise be integrated into thecomputer or other computerized apparatus that is to be controlled by thesensor output.

FIG. 4 further includes a schematic representation of a depth map 410captured by the image sensors 404, in accordance with some embodiments.The depth map, as explained above, comprises a matrix of pixels havingrespective depth values. The pixels 412 corresponding to the hand 406have been segmented out from the background and the wrist in this map.The brightness of each pixel within the depth map 410 correspondsinversely to its depth value, i.e., the measured z distance from theimage sensors 404, with the shade of gray growing darker with increasingdepth. The controller 110 processes these depth values in order toidentify and segment a component of the image (i.e., a group ofneighboring pixels) having characteristics of a human hand. Thesecharacteristics, may include, for example, overall size, shape andmotion from frame to frame of the sequence of depth maps.

FIG. 4 also schematically illustrates a hand skeleton 414 thatcontroller 110 ultimately extracts from the depth map 410 of the hand406, in accordance with some embodiments. In FIG. 4 , the skeleton 414is superimposed on a hand background 416 that has been segmented fromthe original depth map. In some embodiments, key feature points of thehand (e.g., points corresponding to knuckles, finger tips, center of thepalm, end of the hand connecting to wrist, etc.) and optionally on thewrist or arm connected to the hand are identified and located on thehand skeleton 414. In some embodiments, location and movements of thesekey feature points over multiple image frames are used by the controller110 to determine the hand gestures performed by the hand or the currentstate of the hand, in accordance with some embodiments.

FIG. 5 illustrates an example embodiment of the eye tracking device 130(FIG. 1 ). In some embodiments, the eye tracking device 130 iscontrolled by the eye tracking unit 245 (FIG. 2 ) to track the positionand movement of the user's gaze with respect to the scene 105 or withrespect to the CGR content displayed via the display generationcomponent 120. In some embodiments, the eye tracking device 130 isintegrated with the display generation component 120. For example, insome embodiments, when the display generation component 120 is ahead-mounted device such as headset, helmet, goggles, or glasses, or ahandheld device placed in a wearable frame, the head-mounted deviceincludes both a component that generates the CGR content for viewing bythe user and a component for tracking the gaze of the user relative tothe CGR content. In some embodiments, the eye tracking device 130 isseparate from the display generation component 120. For example, whendisplay generation component is a handheld device or a CGR chamber, theeye tracking device 130 is optionally a separate device from thehandheld device or CGR chamber. In some embodiments, the eye trackingdevice 130 is a head-mounted device or part of a head-mounted device. Insome embodiments, the head-mounted eye-tracking device 130 is optionallyused in conjunction with a display generation component that is alsohead-mounted, or a display generation component that is nothead-mounted. In some embodiments, the eye tracking device 130 is not ahead-mounted device, and is optionally used in conjunction with ahead-mounted display generation component. In some embodiments, the eyetracking device 130 is not a head-mounted device, and is optionally partof a non-head-mounted display generation component.

In some embodiments, the display generation component 120 uses a displaymechanism (e.g., left and right near-eye display panels) for displayingframes including left and right images in front of a user's eyes to thusprovide 3D virtual views to the user. For example, a head-mounteddisplay generation component may include left and right optical lenses(referred to herein as eye lenses) located between the display and theuser's eyes. In some embodiments, the display generation component mayinclude or be coupled to one or more external video cameras that capturevideo of the user's environment for display. In some embodiments, ahead-mounted display generation component may have a transparent orsemi-transparent display through which a user may view the physicalenvironment directly and display virtual objects on the transparent orsemi-transparent display. In some embodiments, display generationcomponent projects virtual objects into the physical environment. Thevirtual objects may be projected, for example, on a physical surface oras a holograph, so that an individual, using the system, observes thevirtual objects superimposed over the physical environment. In suchcases, separate display panels and image frames for the left and righteyes may not be necessary.

As shown in FIG. 5 , in some embodiments, a gaze tracking device 130includes at least one eye tracking camera (e.g., infrared (IR) ornear-IR (NIR) cameras), and illumination sources (e.g., IR or NIR lightsources such as an array or ring of LEDs) that emit light (e.g., IR orNIR light) towards the user's eyes. The eye tracking cameras may bepointed towards the user's eyes to receive reflected IR or NIR lightfrom the light sources directly from the eyes, or alternatively may bepointed towards “hot” mirrors located between the user's eyes and thedisplay panels that reflect IR or NIR light from the eyes to the eyetracking cameras while allowing visible light to pass. The gaze trackingdevice 130 optionally captures images of the user's eyes (e.g., as avideo stream captured at 60-120 frames per second (fps)), analyze theimages to generate gaze tracking information, and communicate the gazetracking information to the controller 110. In some embodiments, twoeyes of the user are separately tracked by respective eye trackingcameras and illumination sources. In some embodiments, only one eye ofthe user is tracked by a respective eye tracking camera and illuminationsources.

In some embodiments, the eye tracking device 130 is calibrated using adevice-specific calibration process to determine parameters of the eyetracking device for the specific operating environment 100, for examplethe 3D geometric relationship and parameters of the LEDs, cameras, hotmirrors (if present), eye lenses, and display screen. Thedevice-specific calibration process may be performed at the factory oranother facility prior to delivery of the AR/VR equipment to the enduser. The device-specific calibration process may an automatedcalibration process or a manual calibration process. A user-specificcalibration process may include an estimation of a specific user's eyeparameters, for example the pupil location, fovea location, opticalaxis, visual axis, eye spacing, etc. Once the device-specific anduser-specific parameters are determined for the eye tracking device 130,images captured by the eye tracking cameras can be processed using aglint-assisted method to determine the current visual axis and point ofgaze of the user with respect to the display, in accordance with someembodiments.

As shown in FIG. 5 , the eye tracking device 130 (e.g., 130A or 130B)includes eye lens(es) 520, and a gaze tracking system that includes atleast one eye tracking camera 540 (e.g., infrared (IR) or near-IR (NIR)cameras) positioned on a side of the user's face for which eye trackingis performed, and an illumination source 530 (e.g., IR or NIR lightsources such as an array or ring of NIR light-emitting diodes (LEDs))that emit light (e.g., IR or NIR light) towards the user's eye(s) 592.The eye tracking cameras 540 may be pointed towards mirrors 550 locatedbetween the user's eye(s) 592 and a display 510 (e.g., a left or rightdisplay panel of a head-mounted display, or a display of a handhelddevice, a projector, etc.) that reflect IR or NIR light from the eye(s)592 while allowing visible light to pass (e.g., as shown in the topportion of FIG. 5 ), or alternatively may be pointed towards the user'seye(s) 592 to receive reflected IR or NIR light from the eye(s) 592(e.g., as shown in the bottom portion of FIG. 5 ).

In some embodiments, the controller 110 renders AR or VR frames 562(e.g., left and right frames for left and right display panels) andprovide the frames 562 to the display 510. The controller 110 uses gazetracking input 542 from the eye tracking cameras 540 for variouspurposes, for example in processing the frames 562 for display. Thecontroller 110 optionally estimates the user's point of gaze on thedisplay 510 based on the gaze tracking input 542 obtained from the eyetracking cameras 540 using the glint-assisted methods or other suitablemethods. The point of gaze estimated from the gaze tracking input 542 isoptionally used to determine the direction in which the user iscurrently looking.

The following describes several possible use cases for the user'scurrent gaze direction, and is not intended to be limiting. As anexample use case, the controller 110 may render virtual contentdifferently based on the determined direction of the user's gaze. Forexample, the controller 110 may generate virtual content at a higherresolution in a foveal region determined from the user's current gazedirection than in peripheral regions. As another example, the controllermay position or move virtual content in the view based at least in parton the user's current gaze direction. As another example, the controllermay display particular virtual content in the view based at least inpart on the user's current gaze direction. As another example use casein AR applications, the controller 110 may direct external cameras forcapturing the physical environment of the CGR experience to focus in thedetermined direction. The autofocus mechanism of the external camerasmay then focus on an object or surface in the environment that the useris currently looking at on the display 510. As another example use case,the eye lenses 520 may be focusable lenses, and the gaze trackinginformation is used by the controller to adjust the focus of the eyelenses 520 so that the virtual object that the user is currently lookingat has the proper vergence to match the convergence of the user's eyes592. The controller 110 may leverage the gaze tracking information todirect the eye lenses 520 to adjust focus so that close objects that theuser is looking at appear at the right distance.

In some embodiments, the eye tracking device is part of a head-mounteddevice that includes a display (e.g., display 510), two eye lenses(e.g., eye lens(es) 520), eye tracking cameras (e.g., eye trackingcamera(s) 540), and light sources (e.g., light sources 530 (e.g., IR orNIR LEDs), mounted in a wearable housing. The Light sources emit light(e.g., IR or NIR light) towards the user's eye(s) 592. In someembodiments, the light sources may be arranged in rings or circlesaround each of the lenses as shown in FIG. 5 . In some embodiments,eight light sources 530 (e.g., LEDs) are arranged around each lens 520as an example. However, more or fewer light sources 530 may be used, andother arrangements and locations of light sources 530 may be used.

In some embodiments, the display 510 emits light in the visible lightrange and does not emit light in the IR or NIR range, and thus does notintroduce noise in the gaze tracking system. Note that the location andangle of eye tracking camera(s) 540 is given by way of example, and isnot intended to be limiting. In some embodiments, a single eye trackingcamera 540 located on each side of the user's face. In some embodiments,two or more NIR cameras 540 may be used on each side of the user's face.In some embodiments, a camera 540 with a wider field of view (FOV) and acamera 540 with a narrower FOV may be used on each side of the user'sface. In some embodiments, a camera 540 that operates at one wavelength(e.g. 850 nm) and a camera 540 that operates at a different wavelength(e.g. 940 nm) may be used on each side of the user's face.

Embodiments of the gaze tracking system as illustrated in FIG. 5 may,for example, be used in computer-generated reality (e.g., includingvirtual reality, and/or mixed reality) applications to providecomputer-generated reality (e.g., including virtual reality, augmentedreality, and/or augmented virtuality) experiences to the user.

FIG. 6 illustrates a glint-assisted gaze tracking pipeline, inaccordance with some embodiments. In some embodiments, the gaze trackingpipeline is implemented by a glint-assisted gaze tracing system (e.g.,eye tracking device 130 as illustrated in FIGS. 1 and 5 ). Theglint-assisted gaze tracking system may maintain a tracking state.Initially, the tracking state is off or “NO”. When in the trackingstate, the glint-assisted gaze tracking system uses prior informationfrom the previous frame when analyzing the current frame to track thepupil contour and glints in the current frame. When not in the trackingstate, the glint-assisted gaze tracking system attempts to detect thepupil and glints in the current frame and, if successful, initializesthe tracking state to “YES” and continues with the next frame in thetracking state.

As shown in FIG. 6 , the gaze tracking cameras may capture left andright images of the user's left and right eyes. The captured images arethen input to a gaze tracking pipeline for processing beginning at 610.As indicated by the arrow returning to element 600, the gaze trackingsystem may continue to capture images of the user's eyes, for example ata rate of 60 to 120 frames per second. In some embodiments, each set ofcaptured images may be input to the pipeline for processing. However, insome embodiments or under some conditions, not all captured frames areprocessed by the pipeline.

At 610, for the current captured images, if the tracking state is YES,then the method proceeds to element 640. At 610, if the tracking stateis NO, then as indicated at 620 the images are analyzed to detect theuser's pupils and glints in the images. At 630, if the pupils and glintsare successfully detected, then the method proceeds to element 640.Otherwise, the method returns to element 610 to process next images ofthe user's eyes.

At 640, if proceeding from element 410, the current frames are analyzedto track the pupils and glints based in part on prior information fromthe previous frames. At 640, if proceeding from element 630, thetracking state is initialized based on the detected pupils and glints inthe current frames. Results of processing at element 640 are checked toverify that the results of tracking or detection can be trusted. Forexample, results may be checked to determine if the pupil and asufficient number of glints to perform gaze estimation are successfullytracked or detected in the current frames. At 650, if the results cannotbe trusted, then the tracking state is set to NO and the method returnsto element 610 to process next images of the user's eyes. At 650, if theresults are trusted, then the method proceeds to element 670. At 670,the tracking state is set to YES (if not already YES), and the pupil andglint information is passed to element 680 to estimate the user's pointof gaze.

FIG. 6 is intended to serves as one example of eye tracking technologythat may be used in a particular implementation. As recognized by thoseof ordinary skill in the art, other eye tracking technologies thatcurrently exist or are developed in the future may be used in place ofor in combination with the glint-assisted eye tracking technologydescribe herein in the computer system 101 for providing CGR experiencesto users, in accordance with various embodiments.

In the present disclosure, various input methods are described withrespect to interactions with a computer system. When an example isprovided using one input device or input method and another example isprovided using another input device or input method, it is to beunderstood that each example may be compatible with and optionallyutilizes the input device or input method described with respect toanother example. Similarly, various output methods are described withrespect to interactions with a computer system. When an example isprovided using one output device or output method and another example isprovided using another output device or output method, it is to beunderstood that each example may be compatible with and optionallyutilizes the output device or output method described with respect toanother example. Similarly, various methods are described with respectto interactions with a virtual environment or a mixed realityenvironment through a computer system. When an example is provided usinginteractions with a virtual environment and another example is providedusing mixed reality environment, it is to be understood that eachexample may be compatible with and optionally utilizes the methodsdescribed with respect to another example. As such, the presentdisclosure discloses embodiments that are combinations of the featuresof multiple examples, without exhaustively listing all features of anembodiment in the description of each example embodiment.

User Interfaces and Associated Processes

Attention is now directed towards embodiments of user interfaces (“UI”)and associated processes that may be implemented on a computer system,such as portable multifunction device or a head-mounted device, with adisplay generation component, one or more input devices, and(optionally) one or cameras.

FIGS. 7A-7P illustrate three-dimensional environments displayed via adisplay generation component (e.g., a display generation component 7100,display generation component 7200, a display generation component 120,etc.) and interactions that occur in the three-dimensional environmentcaused by user inputs directed to the three-dimensional environmentand/or inputs received from other computer systems and/or sensors. Insome embodiments, the inputs are directed to a virtual object within thethree-dimensional environment by a user's gaze detected at the positionsof the virtual object, by a hand gesture performed at a location in thephysical environment that corresponds to the position of the virtualobject, by a hand gesture that is performed at a location in thephysical environment that is independent of the position of the virtualobject while the virtual object has input focus (e.g., selected by aconcurrently and/or previously detected gaze input, selected by aconcurrently or previously detected pointer input, selected by aconcurrently and/or previously detected gesture input, etc.), by a inputdevice that has positioned a focus selector object (e.g., a pointerobject, selector object, etc.) at the position of the virtual object,etc. In some embodiments, the inputs are directed to a representation ofa physical object or a virtual object that corresponds to a physicalobject by the user's hand movement (e.g., whole hand movement, wholehand movement in a respective posture, movement of one portion of handrelative to another portion of the hand, relative movement between twohands, etc.) and/or manipulation with respect to the physical object(e.g., touching, swiping, tapping, opening, moving toward, movingrelative to, etc.). In some embodiments, the computer system displayschanges the three-dimensional environment (e.g., displaying additionalvirtual content, or ceasing to display existing virtual content,transitioning between different levels of immersion with which visualcontent is being displayed, etc.) in accordance with inputs from sensors(e.g., image sensors, temperature sensors, biometric sensors, motionsensors, proximity sensors, etc.) and contextual conditions (e.g.,location, time, presence of others in the environment, etc.). In someembodiments, the computer system displays changes the three-dimensionalenvironment (e.g., displaying additional virtual content, or ceasing todisplay existing virtual content, transitioning between different levelsof immersion with which visual content is being displayed, etc.) inaccordance with inputs from other computers used by other users that aresharing the computer-generated environment with the user of the computersystem (e.g., in a shared computer-generated experience, in a sharedvirtual environment, in a shared virtual or augmented realityenvironment of a communication session, etc.).

In some embodiments, the three-dimensional environment that is displayedvia the display generation component is a virtual three-dimensionalenvironment that includes virtual objects and content at differentvirtual positions in the three-dimensional environment without arepresentation of the physical environment. In some embodiments, thethree-dimensional environment is a mixed reality environment thatdisplays virtual objects at different virtual positions in thethree-dimensional environment that are constrained by one or morephysical aspects of the physical environment (e.g., positions andorientations of walls, floors, surfaces, direction of gravity, time ofday, etc.). In some embodiments, the three-dimensional environment is anaugmented reality environment that includes a representation of thephysical environment. The representation of the physical environmentincludes respective representations of physical objects and surfaces atdifferent positions in the three-dimensional environment, such that thespatial relationships between the different physical objects andsurfaces in the physical environment are reflected by the spatialrelationships between the representations of the physical objects andsurfaces in the three-dimensional environment. When virtual objects areplaced relative to the positions of the representations of physicalobjects and surfaces in the three-dimensional environment, they appearto have corresponding spatial relationships with the physical objectsand surfaces in the physical environment. In some embodiments, thecomputer system transitions between displaying the different types ofenvironment (e.g., transitions between presenting a computer-generatedenvironment or experience with different levels of immersion, adjustingthe relative prominence of audio/visual sensory inputs from the virtualcontent and from the representation of the physical environment, etc.)based on user inputs and/or contextual conditions.

In some embodiments, the display generation component includes apass-through portion in which the representation of the physicalenvironment is displayed. In some embodiments, the pass-through portionis a transparent or semi-transparent (e.g., a see-through) portion ofthe display generation component revealing at least a portion ofphysical environment surrounding and within the field of view of user.For example, the pass-through portion is a portion of a head-mounteddisplay or heads-up display that is made semi-transparent (e.g., lessthan 50%, 40%, 30%, 20%, 15%, 10%, or 5% of opacity) or transparent,such that the user can see through it to view the real world surroundingthe user without removing the head-mounted display or moving away fromthe heads-up display. In some embodiments, the pass-through portiongradually transitions from semi-transparent or transparent to fullyopaque when displaying a virtual or mixed reality environment. In someembodiments, the pass-through portion of the display generationcomponent displays a live feed of images or video of at least a portionof physical environment captured by one or more cameras (e.g., rearfacing camera(s) of the mobile device or associated with thehead-mounted display, or other cameras that feed image data to theelectronic device). In some embodiments, the one or more cameras pointat a portion of the physical environment that is directly in front ofthe user's eyes (e.g., behind the display generation component). In someembodiments, the one or more cameras point at a portion of the physicalenvironment that is not directly in front of the user's eyes (e.g., in adifferent physical environment, or to the side or behind the user).

In some embodiments, when displaying virtual objects at positions thatcorrespond to locations of one or more physical objects in the physicalenvironment (e.g., in a virtual reality environment, a mixed realityenvironment, an augmented reality environment, etc.), at least some ofthe virtual objects are displayed in placed of (e.g., replacing displayof) a portion of the live view (e.g., a portion of the physicalenvironment captured in the live view) of the cameras. In someembodiments, at least some of the virtual objects and content areprojected onto the physical surfaces or empty space in the physicalenvironment and are visible through the pass-through portion of thedisplay generation component (e.g., viewable as part of the camera viewof the physical environment, or through the transparent orsemi-transparent portion of the display generation component, etc.). Insome embodiments, at least some of the virtual objects and content aredisplayed to overlay a portion of the display and blocks the view of atleast a portion of the physical environment visible through thetransparent or semi-transparent portion of the display generationcomponent.

In some embodiments, the display generation component displays differentviews of the three-dimensional environment in accordance with userinputs or movements that changes the virtual position of the viewpointof the currently displayed view of the three-dimensional environmentrelative to the three-dimensional environment. In some embodiments, whenthe three-dimensional environment is a virtual environment, theviewpoint moves in accordance with navigation or locomotion requests(e.g., in-air hand gestures, gestures performed by movement of oneportion of the hand relative to another portion of the hand, etc.)without requiring movement of the user's head, torso, and/or the displaygeneration component in the physical environment. In some embodiments,movement of the user's head and/or torso, and/or the movement of thedisplay generation component or other location sensing elements of thecomputer system (e.g., due to the user holding the display generationcomponent or wearing the HMD, etc.), etc., relative to the physicalenvironment causes corresponding movement of the viewpoint (e.g., withcorresponding movement direction, movement distance, movement speed,and/or change in orientation, etc.) relative to the three-dimensionalenvironment, resulting in corresponding change in the currentlydisplayed view of the three-dimensional environment. In someembodiments, when a virtual object has a preset spatial relationshiprelative to the viewpoint, movement of the viewpoint relative to thethree-dimensional environment would cause movement of the virtual objectrelative to the three-dimensional environment while the position of thevirtual object in the field of view is maintained (e.g., the virtualobject is said to be head locked). In some embodiments, a virtual objectis body-locked to the user, and moves relative to the three-dimensionalenvironment when the user moves as a whole in the physical environment(e.g., carrying or wearing the display generation component and/or otherlocation sensing component of the computer system), but will not move inthe three-dimensional environment in response to the user's headmovement (e.g., the display generation component and/or other locationsensing component of the computer system rotating around a fixedlocation of the user in the physical environment).

In some embodiments, the views of the three-dimensional environmentshown in FIGS. 7A-7P include representation(s) of a user's hand(s),arm(s), and/or wrist(s). In some embodiments, the representation(s) arepart of the representation of the physical environment provided via thedisplay generation component. In some embodiments, the representationsare not part of the representation of the physical environment and areseparately captured (e.g., by one or more camera's pointing toward theuser's hand(s), arm(s), and wrist(s)) and displayed in thethree-dimensional environment independent of the view of thethree-dimensional environment. In some embodiments, therepresentation(s) include camera images as captured by one or morecameras of the computer system(s), or stylized versions of the arm(s),wrist(s) and/or hand(s) based on information captured by varioussensors). In some embodiments, the representation(s) replace display of,are overlaid on, or block the view of, a portion of the representationof the physical environment. In some embodiments, when the displaygeneration component does not provide a view of a physical environment,and provides a completely virtual environment (e.g., no camera view ortransparent pass-through portion), real-time visual representations(e.g., stylize representations or segmented camera images) of one orboth arms, wrists, and/or hands of the user may still be displayed inthe virtual environment.

FIGS. 7A-7C are block diagrams illustrating interaction with a userinterface object in a computer-generated three-dimensional environmentthat is shared between two or more users, in accordance with someembodiments.

In some embodiments, the computer system permits multiple users (e.g.,the first user 7102, the second user 7002, another user, etc.) to havethe right to access a first user interface object (e.g., first userinterface object 7016, another user interface object, a control panel, avirtual menu, a media object, etc.) displayed in a three-dimensionalenvironment (e.g., a three-dimensional environment 7015, another virtualenvironment or augmented reality environment, etc.), but prevents a user(e.g., the first user 7102, or another user different from the firstuser 7102, etc.) from accessing the first user interface object whileanother user (e.g., the second user 7002, another user different fromthe second user 7002, etc.) is interacting with the first user interfaceobject. When displaying a view of the three-dimensional environmentincluding the first user interface object via a first display generationcomponent (e.g., display generation component 7200, a different type ofdisplay generation component such as an HMD, etc.) used by a first user(e.g., the first user 7102), the computer system detects a first userinput (e.g., a gaze input, a hand movement, a combination of a gazeinput and a movement of the user's hand, etc.) that is directed to thefirst user interface object. In response to detecting the first userinput, the computer system, depending whether or not the first userinterface object is available for interaction with the first user at thetime, performs a first operation corresponding to the first user inputwith respect to the first user interface object (e.g., moving the firstuser interface object or a representation thereof to the representation7202′ of the first user's hand 7202, performing a function associatedwith the first user interface object that changes the three-dimensionalenvironment (e.g., causes display or dismissal of virtual content in thethree-dimensional environment, changing other virtual content in thethree-dimensional environment, etc.), etc.), or displays a visualindication that the first user interface object is not available forinteraction with the first user and forgoes performance of the firstoperation. The computer system provides the visual indication andforgoes performance of the first operation in accordance with adetermination that another user (e.g., the second user 7002) has controlof the first user interface object at the time (e.g., another user isinteracting with the first user interface object, is interacting withthe first user interface object in a manner that excludes the firstuser's contemporaneous interaction, and/or has a lock on the first userinterface object for the type of action that the first user isattempting to perform, etc.). In some embodiments, displaying the visualindication includes moving the first user interface object in the viewof the three-dimensional environment shown to the first user to maintaina preset distance between the first user interface object and theapproaching representation of the hand of the first user. In someembodiments, displaying the visual indication includes changing thevisual appearance of the first user interface object in the view of thethree-dimensional environment shown to the first user (e.g., as shown inFIG. 7C, the view on the user 7102 side). In some embodiments, when thefirst user interface object is released to the first user by thecontrolling user (e.g., by a throw gesture, a toss gesture, etc.), thecomputer system rotates the first user interface object such that thefirst user interface object is displayed with a preset orientationrelative to the viewpoint of the currently displayed view of thethree-dimensional environment shown to the first user (e.g., withcontent side or control side facing toward the first user 7102). In someembodiments, the computer system provides controlling access of thefirst user interface object to the first user by displaying arepresentation of the first user interface object at a position at ornear the representation of a portion of the first user (e.g., in therepresentation of the hand 7202 of the first user 7102, within an arm'sreach of the virtual position of the user's face, etc.).

In the example shown in FIGS. 7A-7C, the three-dimensional environment7015 is shared between the first user 7102 and the second user 7002 inresponse to a request that is initiated from one of the users 7102 and7002 using a computer system controlled by said one user, and acceptedby another of the users 7102 and 7002 using a computer system controlledby said another user, in accordance with some embodiments. In someembodiments, both users have received and accepted the request to sharethe three-dimensional environment using their respective computersystems from the computer system used by a third user. In someembodiments, both users have sent requests to share thethree-dimensional environment to a server using their respectivecomputer systems, where their requests were accepted by the server. Whensharing the computer-generated three-dimensional environment, thelocations and orientations of the users and of their respective heads,eyes, hands, arms, and/or wrists are captured in real-time orperiodically by sensors (e.g., cameras, motion sensors, etc.) and thelocation and orientation data is provided to one or both of the computersystems controlled by the users, and/or to a server that is incommunication with the computer systems. The location data is used bythe computer systems and/or server to determine the respective positionsand orientations of the users and of their respective heads, eyes,hands, arms, and/or wrists in the computer-generated three-dimensionalenvironment, and correspondingly, the respective positions of therepresentations of the users including their respective heads, arms,hands, and/or wrists in the views of the three-dimensional environmentprovided via the different display generation components associated withthe users, as well as the viewing perspectives and viewpoints of theviews of the three-dimensional environment provided via the differentdisplay generation components associated with the users.

In some embodiments, when two or more users share a computer-generatedenvironment (e.g., a virtual conference call, a chat session, amulti-player game, a shared computer-generated experience (e.g., groupmeditation, exercise, game, collaborative work, etc.), etc.), they maywish to control and/or manipulate the same user interface object (e.g.,a virtual ball, a virtual control panel, a document or media content, avirtual menu, a user interface, etc.) present in the computer-generatedenvironment. This sometimes creates difficulty for the computer systemto consistently prioritize the different user's actions with respect tothe user interface object and the resulting change in thethree-dimensional environment may be confusing to the users. Asdisclosed herein, the computer system provides visual feedback inresponse to a first user 7102's attempt to interact with a first userinterface object 7016 that is already in the control of a second user7002 in the environment by changing a set of appearance properties ofthe first user interface object in the view 7015-1 of the environmentpresented to the first user 7102, thereby reducing conflict between theactions of the users and reducing user confusion when they interact withthe first user interface object 7016. In some embodiments, the firstuser interface object 7016 presented in the view 7015-2 of thethree-dimensional environment shown to the second user 7002 that hascontrol of the first user interface object is not changed as a result ofthe first user's attempt to interact with the first user interfaceobject, and does not cause distraction to the second user 7002 when thesecond user 7002 interacts with the first user interface object 7016.

FIG. 7A illustrates an exemplary physical environment (e.g., scene 105,another indoor or outdoor physical environment, etc.). In someembodiments, as shown in FIG. 7A, two or more users (e.g., the user7102, the user 7002, etc.) are present in the same physical environment.The first user 7102 is viewing a first view 7015-1 of thethree-dimensional environment 7015 (e.g., an augmented realityenvironment, a virtual environment, etc.) via a first display generationcomponent (e.g., the display generation component 7200, another type ofdisplay generation component such as an HMD, etc. used by the firstuser, etc.). The second user 7002 is viewing a second view 7015-2 of thesame three-dimensional environment 7015 via a second display generationcomponent (e.g., the display generation component 7100, another type ofdisplay generation component such as an HMD, etc. used by the seconduser, etc.). In some embodiments, the three-dimensional environment 7015(e.g., labeled as 7015-1 when presented via the first display generationcomponent 7200, labeled as 7015-2 when presented via the second displaygeneration component 7100, etc.) is an environment of a sharedcomputer-generated experience, a communication session, an applicationenvironment, a game, a movie, etc.

In some embodiments, the first user 7102 and the second user 7002 arenot necessarily located in the same physical environment at the sametime, and may be separately located in two different physicalenvironment. In some embodiments, the three-dimensional environment 7015includes a representation of the physical environment of the first userand not of the second user, and the first user and the second user havea shared experience in the three-dimensional environment based on thephysical environment of the first user. In some embodiments, thethree-dimensional environment 7015 includes a representation of thephysical environment of the second user and not of the first user, andthe first user and the second user have a shared experience in thethree-dimensional environment based on the physical environment of thesecond user. In some embodiments, the three-dimensional environment 7015includes a representation of a third physical environment that is notthe physical environment of the first user or the physical environmentof the second user, and the first user and the second user have a sharedexperience in the three-dimensional environment based on the thirdphysical environment (e.g., the physical environment of a third userthat is participating in the shared experience, another physicalenvironment that is not associated with a user or that is associatedwith a user who is not participating in the shared experience, etc.). Insome embodiments, the three-dimensional environment 7015 includes avirtual three-dimensional environment, and the first user and the seconduser have a shared experience in the virtual three-dimensionalenvironment. In some embodiments, the positions and movements of thefirst user and the second user in their respective physical environments(e.g., same physical environment, different physical environments, etc.)are mapped (e.g., using the same mapping relationship, or differentmapping relationship, etc.) to positions and movements in the samethree-dimensional environment, but the appearance of thethree-dimensional environments may be adjusted (e.g., with differentwallpapers, color schemes, with different virtual furniture, etc.) totailor to a respective user in the view of the three-dimensionalenvironment shown to the respective user.

In some embodiments, the computer system determines that thethree-dimensional environment is at least partially shared between thefirst user 7102 and the second user 7002 in accordance with adetermination that at least a spatial portion of the environment 7015(e.g., a spatial portion of the environment that corresponds to theliving room, but not the kitchen; a spatial portion of the environmentthat corresponds to the portion of physical space in front of the firstuser, but no the portion of physical space behind the first user, etc.)is shared. In some embodiments, the computer system determines that thethree-dimensional environment is at least partially shared between thefirst user and the second user in accordance with a determination thatat least a spatial portion of the environment 7015 is shared during atleast a period of time (e.g., during a communication session between thefirst user and the second user, during the morning, during workinghours, when both users are online, etc.). In some embodiments, thecomputer system determines that the three-dimensional environment 7105is at least partially shared between the first user and the second userin accordance with a determination that the objects in the environment7015 are shared fully or partially (e.g., simultaneously viewable andaccessible, simultaneously viewable but not simultaneously accessible,viewable but not accessible when others have control (e.g., said otherscan be viewing or not viewing the object, etc.). In some embodiments,the computer system determines that the three-dimensional environment7015 is at least partially shared between the first user and the seconduser in accordance with a determination that at least a portion of thethree-dimensional environment 7015 (e.g., the portion shown in the firstview 7015-1 of the three-dimensional environment, another portion of thethree-dimensional environment 7015, etc.) is displayed for viewing byboth the first user and the second user at the same time. In someembodiments, the computer system determines that the three-dimensionalenvironment 7015 is at least partially shared between the first user andthe second user in accordance with a determination that some or all ofthe virtual objects in the three-dimensional environment areconcurrently displayed in the three-dimensional environment to both thefirst user and the second user.

In FIGS. 7B and 7C, the computer system displays the first view 7015-1of the three-dimensional environment 7015 that is at least partiallyshared between the first user 7102 and the second user 7002, via thefirst display generation component 7200; and at substantially the sametime (e.g., adjusted for network delays, processing time delays, etc.),the computer system or another computer system in communication with thecomputer system displays the second view 7015-2 of the three-dimensionalenvironment 7105 via the second display generation component 7100. Thefirst view 7015-1 and the second view 7015-2 both include at least afirst portion of the three-dimensional environment (e.g., a respectiveportion that corresponds to the same portion of the physical environmentrepresented in the three-dimensional environment, a respective portionthat corresponds to the same portion of the virtual environment of thethree-dimensional environment, etc.), in accordance with someembodiment. In some embodiments, the first portion of thethree-dimensional environment is optionally shown from different viewingangles in the first view 7015-1 and the second view 7015-2 of thethree-dimensional environment 7105 (e.g., based on the respectivespatial relationships between the first user and his/her physicalenvironment, and/or the respective spatial relationships between thefirst user and his/her physical environment, etc.)

In some embodiments, the first view 7015-1 has a first viewpoint with aposition that corresponds to the current location of the first user 7102in his/her physical environment, and the position moves in thethree-dimensional environment 7015 in accordance with the movement ofthe first user 7102 in the physical environment of the first user 7102(e.g., scene 105, another physical environment, etc.). In someembodiments, the second view 7015-2 has a second viewpoint with aposition in the three-dimensional environment 7015 that corresponds tothe current location of the second user 7002 in his/her physicalenvironment, and the position moves in the three-dimensional environment7015 in accordance with the movement of the second user 7002 in thephysical environment of the second user (e.g., scene 105, anotherphysical environment, etc.). In some embodiments, the viewpoint of acurrently displayed view of the three-dimensional environment 7015 thatis shown via a respective display generation component (e.g., the firstdisplay generation component 7200, the second display generationcomponent 7100, etc.) has a position in the three-dimensionalenvironment 7015 that corresponds to the current location of therespective display generation component, and the position moves in thethree-dimensional environment 7015 in accordance with the movement ofthe respective display generation component in the physical environmentof the respective display generation component (e.g., scene 105, anotherphysical environment, etc.). In some embodiments, the viewpoint of acurrently displayed view of the three-dimensional environment 7015 thatis shown via a respective display generation component (e.g., the firstdisplay generation component 7200, the second display generationcomponent 7100, etc.) has a position in the three-dimensionalenvironment that corresponds to the current location of one or morecameras associated with the respective display generation component, andthe position moves in the three-dimensional environment 7015 inaccordance with the movement of the one or more cameras associated withthe respective display generation component in the physical environmentof the respective display generation component (e.g., scene 105, anotherphysical environment, etc.). In the example shown in FIGS. 7A-7C, eventhough the first view 7015-1 and the second view 7015-2 appear to havethe same viewpoint, it is to be understood that the respective viewsshown via the first display generation component 7200 and the seconddisplay generation 7100 component and their corresponding viewpoints areseparately and independently determined based on the spatialrelationships and movements existing in the respective physicalenvironments of the first display generation component (and the firstuser) and the second display generation component (and second user), anddo not have to be exactly the same at a given time.

In FIGS. 7B and 7C, the first view 7015-1 and the second view 7015-2 ofthe three-dimensional environment 7015 include one or more userinterface objects (e.g., the first user interface object 7016, a seconduser interface object 7018, other user interface objects, virtualthree-dimensional objects, etc.), and optionally, one or more surfaces(e.g., representations 7004′ or 7004″ of the wall 7004, representations7006′ or 7006″ of the wall 7006, representation 7008′ or 7008″ of thefloor 7008, virtual surfaces such as virtual walls, virtual screens,virtual windows, virtual scenery, etc.), and/or representations of oneor more physical objects (e.g., representation 7014′ or 7014″ of aphysical object 7014 in the physical environment 7014, representationsof other physical objects in another physical environment represented inthe three-dimensional environment 7015, etc.). In some embodiments, thefirst view 7015-1 and the second view 7015-2 do not include arepresentation of a physical environment and includes a virtualthree-dimensional environment (e.g., a virtual conference room, a gameenvironment, a virtual experience, a virtual sports arena, etc.).

In some embodiments, the first user interface object 7016 is arepresentation of an application, and interaction with the first userinterface object that meets preset criteria causes the computer systemto start the application in the three-dimensional environment or performan application function of the application. In some embodiments, thefirst user interface object 7016 is a user interface that includes aplurality of user interface objects (e.g., selectable avatars,selectable menu items, selectable device controls, selectable contentitems, slider controls, buttons, etc.). In some embodiments, the firstuser interface object 7016 is a virtual three-dimensional object thatcan be manipulated (e.g., deformed, separated into parts, rotated,moved, etc.) in the three-dimensional environment in accordance with theuser's hand movement in the physical environment. In some embodiments,the first user interface object 7016 is a single control or a controlpanel that includes multiple controls corresponding to differentfunctions or operations. In some embodiments, the first user interfaceobject 7016 is an information item, a notification, an alert, etc. Insome embodiments, the first user interface object 7016 is a media itemor a document, etc.

In some embodiments, as shown in FIGS. 7B and 7C, the first view 7015-1includes a representation 7202′ of a hand 7202 of the first user 7102,and a representation 7028′ of a hand 7028 of the second user 7002; andthe second view 7015-2 includes a representation 7202″ of the hand 7202of the first user 7102, and a representation 7028″ of the hand 7028 ofthe second user 7002. In the scenario shown in FIGS. 7B and 7C, thesecond user 7002 has control of the first user interface object 7016 inexclusion of contemporaneous interaction between the first user 7102 andthe first user interface object 7016. For example, in some embodiments,when the first user interface object 7016 is in the control of the firstuser 7002, the first user interface object 7016 is displayed at aposition in the three-dimensional environment 7015 that corresponds to alocation of the hand 7028 of the second user 7002 in the physicalenvironment of the second user 7002. In some embodiments, when the firstuser interface object 7016 is in the control of the second user 7002, arepresentation of the first user interface object 7016 is displayed at aposition in the three-dimensional environment 7015 that corresponds tothe location of the hand 7028 of the second user 7002 in the physicalenvironment of the second user 7002, while the first user interfaceobject 7016 is displayed at another position that is separate from theposition of the representation of the first user interface object 7016.In this example, the second user 7002 has control of the first userinterface object 7016, and the first user interface object 7016 isdisplayed at a position in the three-dimensional environment 7015 thatcorresponds to the location of the second user's hand 7028. In someembodiments, when the first user interface object 7016 is in the controlof the second user 7002, the first user interface object 7016 isoriented in the three-dimensional environment 7105 such that a presetsurface (e.g., a front surface A, a content presenting surface, aninteractive surface, etc.) of the first user interface object 7016 facestoward the viewpoint corresponding to the currently displayed secondview 7015-2 of the three-dimensional environment 7015 (e.g., the viewthat is shown to the second user 7002 who has control of the first userinterface object 7016, the view that is displayed by the second displaygeneration component 7100, etc.). In some embodiments, the first userinterface object 7016 can be reoriented in the three-dimensionalenvironment by the second user 7002 who has control of the first userinterface object 7016, such that the preset surface of the first userinterface object 7016 faces toward the viewpoint corresponding to thecurrently displayed first view 7015-1 of the three-dimensionalenvironment (e.g., the view that is shown to the first user 7102 whodoes not have control of the first user interface object 7016 at thetime, the view that is displayed by the first display generationcomponent 7200, etc.). In some embodiments, at least some of the contenton the first user interface object 7016 is only shown in the second view7015-2 of the three-dimensional environment, and not show in the firstview 7015-1 of the three-dimensional environment, when the first userinterface object 7016 is in the control of the second user 7002 and notshared with the first user 7102 (e.g., even if the content displayingside of the first user interface object 7016 is within the first view7015-1 of the three-dimensional environment that is presented to thefirst user 7102 by the first display generation component 7200). In someembodiments, the second user 7002 can make the hidden content of thefirst user interface object 7016 visible to the first user 7102 byre-orientating the first user interface object 7016, such that thecontent presenting side of the first user interface object 7016 is facedaway from the viewpoint of the second view 7015-2.

In FIGS. 7B and 7C, the first view 7015-1 and the second view 7015-2both include a respective representation (e.g., representation 7202′ or7202″) of the first user's hand 7202 and a respective representation(e.g., representation 7028′ or 7028″) of the second user's hand 7028. Insome embodiments, the computer system displays the representations ofthe hands based on camera views of the users' hands. In someembodiments, the computer system provides a view of the representationsof the hands through a transparent portion of the display generationcomponent(s). In some embodiments, the computer system generatesstylistic representations of the user's hands based on sensorinformation received from one or more sensors located in the physicalenvironment(s) of the first user and the second user. In someembodiments, the position and configuration of the representations ofthe user's hand(s) change in accordance with the location(s) andconfiguration(s) of the user's hand(s) in the physical environment(s) ofthe users. In some embodiments, the computer system displays therepresentations of the hands based on camera views of the users' hands.In some embodiments, the computer system displays the representation ofthe hand of one user, but not the representation of the hand of theother user, at a given time. For example, when the second user 7002 hascontrol of the first user interface object 7016, the representation ofthe second user's hand 7028 is, optionally, displayed only in the secondview 7015-2 shown to the second user 7002, and not in the first view7015-1 shown to the first user 7102. In another example, therepresentation of a user's hand may move in and out of the field of viewprovided via a respective display generation component, due to themovements of the first user and/or the second user (and/or theirrespective display generation components or cameras, etc.) in theirrespective physical environments.

In FIGS. 7B and 7C, in the second view 7015-2 of the three-dimensionalenvironment that is displayed via the second display generationcomponent 7100 used by the second user 7002 who has control of the firstuser interface object 7016, the first user interface object 7016 isdisplayed with a first set of appearance properties (e.g., the normalappearance (e.g., first shape, first size, first color, first opacity,first level of saturation, first level of luminance, etc.) of the firstuser interface object as displayed by the second display generationcomponent to the second user). The first user interface object 7016maintains the first set of appearance properties in the control of thesecond user 7002, irrespective of whether or not the first user 7102 isattempting to access the first user interface object 7016 with arespective movement or input directed to the first user interfaceobject. The first user interface object 7016 may change its appearancein a respective way in accordance with the interaction between thesecond user 7002 and the first user interface object 7016 through thecomputer system used by the second user 7002. These changes in theappearance caused by the interaction between the second user 7002 andthe first user interface object 7016 are optionally shown in both thefirst view 7015-1 and the second view 7015-2 at any given time that thechanges occur.

In FIG. 7B, when the first user 7002 is not attempting to access or gaincontrol of the first user interface object 7016 (e.g., via movement of aportion of the user such as the hand of the user, via a gaze input, viaan in-air gesture, via a gesture that involves movement of one portionof a hand relative to another portion of the hand, via an input providedvia a control object, etc.) while the first user interface object 7016is in the control of the second user 7002, the first user interfaceobject 7016 is displayed with the same first set of appearanceproperties in the first view 7015-1 as in the second view 7015-2 of thethree-dimensional environment (optionally, from a different viewingperspective, and/or with redaction of hidden content, etc.). Themovement of the first user's hand 7202 in the physical environment ofthe first user 7102 may be represented in both the first view 7015-1 andthe second view 7015-2 if the first view and the second view bothcaptures the portion of the three-dimensional environment thatcorresponds to the location of the physical space that includes thefirst user's hand 7202.

In contrast, in FIG. 7C, the computer system detects a first user inputprovided by the first user 7102 that is directed to the first userinterface object 7016. For example, in some embodiments, the computersystem detects movement of a portion of the first user 7102 (e.g., theuser's hand 7202, another hand of the first user, etc.) to a location inthe physical environment of the first user 7102 that corresponds to theposition of the first user interface object 7016 in thethree-dimensional environment 7015. In some embodiments, the computersystem detects a gaze input directed to the first user interface object7016 and a control input (e.g., a finger movement gesture, an in airgesture, an input provided by a controller, etc.) that is detected inconjunction with the gaze input. In the example shown in FIG. 7C, thefirst user input is movement of the first user's hand 7202 to a locationcorresponding to the position of the first user interface object 7016,and, optionally, with a movement or posture to grab the first userinterface object 7016 in the three-dimensional environment. In someembodiments, the representation of the movement, position, and/orposture of the hand 7202 of the first user 7102 is shown in both thefirst view 7015-1 and the second view 7015-2. In some embodiments, therepresentation of the movement, position, and/or posture of the hand7202 of the first user 7102 is shown in only the first view 7015-1 andnot in the second view 7015-2. In some embodiments, by not showing themovement, position, and/or posture of the hand 7202 of the first user7102 in the second view 7015-2, the computer system used by the seconduser 7002 reduces the distraction to the second user 7002 when thesecond user 7002 interacts with the first user interface object 7016.

In FIG. 7C, in response to detecting the first user input that isdirected to the first user interface object 7016 and in accordance witha determination that the second user 7002 is currently interacting withthe first user interface object (e.g., has control of the first userinterface object 7016, has control of the first user interface object inexclusion of a requested interaction by the first user 7102, etc.), thecomputer system displays a visual indication that the first userinterface object 7016 is not available for interaction with the firstuser 7102. In some embodiments, displaying the visual indicationincludes changing at least one of an appearance of the first userinterface object 7016 or a position of the first user interface object7016 in the first view 7015-1 of the three-dimensional environment 7015.

In some embodiments, the computer system determines that the second user7002 is currently interacting with the first user interface object 7016in accordance with a determination that the first user interface object7016 has a preset spatial relationship to a virtual position of thesecond user 7002 in the three-dimensional environment (e.g., the firstuser interface object 7016 is in the representation of the second user'spalm or hand 7028, the first user interface object 7016 is within thesecond user's private space that is within the first view 7015-1 of thethree-dimensional environment, etc.). In some embodiments, the computersystem determines that the second user 7002 is currently interactingwith the first user interface object 7016 in accordance with adetermination that the second user 7002 is controlling, selecting,moving, modifying, and/or otherwise interacting with the first userinterface object 7016 through a computer system that displays the secondview 7015-2 of the three-dimensional environment via the second displaygeneration component 7100.

In some embodiments, to display the visual indication in the first view7015-1 of the three-dimensional environment 7015 to indicate that thefirst user interface object 7016 is not available for interaction withthe first user 7102, the computer system displays the first userinterface object 7016 with a second set of appearance properties (e.g.,second shape, second size, second color, second opacity, second level ofsaturation, second level of luminance, etc.) that are different from thefirst set of appearance properties (e.g., the second set of appearanceproperties provide a visual indication that the first user interfaceobject is in control of the second user at this moment, and is notavailable for interacting with the first user). For example, the firstuser interface object 7016 shown in the first view 7015-1 in FIG. 7C ismore translucent than that shown in the second view 7015-2 in FIG. 7C.In some embodiments, to display the visual indication in the first view7015-1 of the three-dimensional environment 7015 to indicate that thefirst user interface object 7016 is not available for interaction withthe first user 7102, the computer system moves the first user interfaceobject 7016 out of the way when the first user 7102 tries to grab it. Insome embodiments, the first user interface object 7016 maintains itsappearance and/or position in the second view 7015-2 displayed to thesecond user 7002, as the visual indication only needs to be displayed tothe first user 7102. In some embodiments, if the first user inputprovided by the first user 7102 corresponds to a request to perform afirst operation with respect to the first user interface object 7016,the computer system, in accordance with a determination that the seconduser 7002 is currently interacting with the first user interface object7016 (e.g., has control of the first user interface object 7016, hascontrol of the first user interface object in exclusion of a requestedinteraction by the first user 7102, etc.), does not perform the firstoperation with respect to the first user interface object 7016. Forexample, in some embodiments, the computer system does not show thefirst user interface object 7106 being grabbed by the representation7202′ of the first user's hand 7202. In some embodiments, the computersystem does not show a ghost image or another representation of thefirst user interface object 7016 moving into the representation 7202′ ofthe first user's hand 7202.

In some embodiments, in response to detecting the first user input thatis directed to the first user interface object 7106 and in accordancewith a determination that the second user 7002 is not currentlyinteracting with the first user interface object 7016, the computersystem performs the first operation with respect to the first userinterface object in accordance with the first user input. In someembodiments, performing the first operation includes showing the firstuser interface object 7016 being grabbed or moved by the first user 7102in accordance with the first user input (e.g., moved toward a virtualposition of the first user 7102 in the three-dimensional environment,moved in accordance with the movement of the first user input, etc.). Insome embodiments, performing the first operation includes showing aghost image or other representation of the first user interface object7016 being grabbed and/or moving into a representation 7202′ of thefirst user's hand 7202. In some embodiments, the first user interfaceobject 7106 continues to be displayed with the first set of appearanceproperties (e.g., at its original location or in a representation of thefirst user's hand, etc.) in accordance with a determination that thesecond user 7002 was not interacting with the first user interfaceobject 7016 when the first user input from the first user 7102 wasdetected.

In some embodiments, when the first user 7102 attempts to grab the firstuser interface object 7016 or otherwise interact with the first userinterface object while the second user 7002 is interacting with thefirst user interface object, the computer system changes the appearanceof the first user interface object, such as fading out the first userinterface object in the first view 7015-1 displayed to the first user7102 as the first user 7102 tries to grab the first user interfaceobject 7016. For example, the computer system changes at least one ofthe first set of appearance properties of the first user interfaceobject 7016 (e.g., increasing a transparency level, reducing colorsaturation, reducing opacity, blurring, darkening, reducing resolution,shrinking in size, etc. of the first user interface object, optionally,while maintaining the appearance of the surrounding environment of thefirst user interface object 7016 (e.g., not changing the appearanceand/or visual prominence of the surrounding environment), etc.) toreduce visual prominence of the first user interface object 7016 in thefirst view 7015-1 of the three-dimensional environment. In someembodiments, in response to detecting that the first user 7102 hasceased to attempt to interact with the first user interface object 7016,the computer system restores (e.g., to the level existed immediatelyprior to detecting the first user input, or prior to changes being madein response to detecting the first user input, etc.) at least one (e.g.,some, all, etc.) of the first set of appearance properties of the firstuser interface object that was changed in response to the first user'sattempts to grab the first user interface object or otherwise interactwith the first user interface object, to restore the visual prominenceof the first user interface object.

In some embodiments, if the first user interface object 7016 is movedaway from the position that corresponds to the location of the firstuser's hand 7202 (e.g., moved away from the representation 7202′ of thehand 7202 in the three-dimensional environment 7015 by the action of thesecond user 7002, and/or in accordance with other events that occurredin the three-dimensional environment (e.g., events that are unrelated tothe attempt for interaction by the first user 7102), etc.), the computersystem restores (e.g., to the level existed immediately prior todetecting the first user input, or prior to changes being made inresponse to detecting the first user input, etc.) at least one of (e.g.,some of, all of, etc.) the first set of appearance properties of thefirst user interface object that was changed in response to the firstuser's attempt to grab the first user interface object or otherwiseinteract with the first user interface object, to restore the visualprominence of the first user interface object.

In some embodiments, after the visual indication that the first userinterface object 7016 is not available for interaction with the firstuser 7102 is displayed in the first view 7015-1, the computer systemcontinues to display the visual indication until the computer systemdetects that the second user 7002 is no longer interacting with thefirst user interface object and/or has relinquished control of the firstuser interface object, such that the first user interface object isavailable for interaction with the first user 7102. In some embodiments,after the visual indication that the first user interface object 7016 isnot available for interaction with the first user 7102 is displayed inthe first view 7015-1, the computer system continues to display thevisual indication for a preset period of time (e.g., ten seconds, fiveseconds, etc.) after the first user has ceased to attempt to interactwith the first user interface object 7106 via the first user input oranother input.

In some embodiments, the first user interface object 7016 can be sent toa position that corresponds to the location of the first user (e.g., aposition that corresponds to the hand 7202 of the first user 7102, aposition that corresponds to a private space surrounding the first user7102, etc.) in accordance with a gesture input (e.g., a toss gesture, athrow gesture, a push gesture, etc.) provided by the second user 7002who has control of the first user interface object 7016. In someembodiments, the first user interface object 7016 rotates (e.g.,reorients, changes a facing direction, etc.) while traveling from afirst position to a second position in the three-dimensional environment7015 as a result of the gesture input provided by the second user 7002.In some embodiments, the first user interface object 7016 can also besent to a position that corresponds to the location of the second user7002 in accordance with a gesture input (e.g., a toss gesture, a throwgesture, a push gesture, etc.) provided by the first user 7102 after thefirst user 7102 has gained control of the first user interface object7016. In some embodiments, the first user interface object 7016 rotates(e.g., reorients, changes a facing direction, etc.) while traveling fromthe second position to a third position in the three-dimensionalenvironment 7015 as a result of the gesture input provided by the firstuser 7102. In some embodiments, the first user interface object 7106rotates to have its content presenting side or interactive side facingtoward the recipient of the first user interface object.

In some embodiments, the first user interface object 7016 can be sent toa position in the three-dimensional environment where the first userinterface object can be seen by both the first user and the second userwith a better view (e.g., displayed in the center of thethree-dimensional environment 7015, displayed at a position thatcorresponds to a wall of the physical environment 105, displayed at avirtual surface in the three-dimensional environment 7015, etc.) inresponse to a gesture input (e.g., a toss gesture, a throw gesture, apush gesture, etc.) provided by the user who has control of the firstuser interface object. In some embodiments, the first user interfaceobject rotates (e.g., reorients, changing a facing direction, etc.)while traveling to the position in the three-dimensional environment,such that when it arrives at the position in the three-dimensionalenvironment, it will have an orientation that enables both the firstuser and the second user to view its content and/or interactive sideand/or have a preset spatial relationship (e.g., overlaying, parallelto, at an angle relative to, perpendicular to, upright relative to,etc.) to a surface (e.g., a representation of a wall surface, tablesurface, a virtual surface, a virtual screen, a virtual tabletop, etc.)at the position of the three-dimensional environment.

In some embodiments, the computer system changes the position of thefirst user interface object 7016 in the first view 7015-1 of thethree-dimensional environment as the visual indication that the firstuser interface object 7016 is not available for interaction with thefirst user 7102. In some embodiments, changing the position of the firstuser interface object in the first view 7015-1 of the three-dimensionalenvironment includes moving the first user interface object 7016 fromthe original position of the first user interface object to maintain atleast a preset distance between the first user interface object and arepresentation 7202′ of the hand 7202 of the first user 7102 thatprovided the first user input (e.g., the first user interface objectappears to move in one or more directions to avoid the representation7202′ of the hand 7202 of the first user 7102 that tries to grab thefirst user interface object). In some embodiments, the movement of thefirst user interface object 7016 is accompanied by changes made to theappearance of the first user interface object (e.g., the first userinterface object appears to be faded or dimmed while moving to avoid therepresentation 7202′ of the hand of the first user 7102 getting tooclose to itself).

In some embodiments, if the first user interface object 7016 is not inthe control of the second user 7002, and is available for interactionwith the first user 7102, the computer system moves the first userinterface object 7016 toward the representation 7202′ of the firstuser's hand 7202 in the first view 7015-1 of the three-dimensionalenvironment 7015, and optionally, also in the second view 7015-2 of thethree-dimensional environment.

In some embodiments, the first user input provided by the first user7102 includes (e.g., is, includes, starts with, ends with, etc.) apredefined selection gesture (e.g., the selection gesture is a pinchgesture that includes touch-down of an index finger on a thumb of thesame hand (optionally, followed by lifting off of the index finger fromthe thumb, or flick of the wrist connected to the hand, or translationof the whole hand, etc.), a gesture that includes an index finger and athumb of the same hand pulling apart from each other from a touchingposture, a pinch gesture, a pinch and drag gesture, a pinch and flickgesture, etc.). In some embodiments, the computer system selects thefirst user interface object 7016 as a target for a subsequent input(e.g., a drag gesture while the pinch gesture is maintained, a flickgesture while the pinch gesture is maintained, a drag gesture after thepredefined selection gesture is terminated, etc.) received from thefirst user 7102, in response to detecting the first user input while thesecond user 7002 is not interacting with the first user interface object7016. In some embodiments, in conjunction with selecting the first userinterface object 7016 as a target for a subsequent input received fromthe first user 7102, the computer system displays a representation ofthe first user interface object 7016 (e.g., a duplicate of the firstuser interface object, a ghost image of the first user interface object,etc.) at a position that corresponds to a location of the hand 7202 ofthe first user 7102, while maintaining the first user interface object7106 at the first position in the first view 7015-1 of thethree-dimensional environment (e.g., the first user interface objectremains at its original location, but can be “remotely” controlled bythe first user 7102 in accordance with interaction between the firstuser 7102 and the representation of the first user interface object). Insome embodiments, the representation of the first user interface objectis displayed near the representation 7202′ of the first user's hand7202, but does not go to the position that corresponds to the locationof the first user's hand until the computer system detects anotherselection input provided by the first user 7102. In some embodiments,the computer system changes the shape of the representation of the firstuser interface object in accordance with a determination that the firstuser 7102 is providing an input that is consistent with the requirementsof the selection input, and the change in the shape of therepresentation of the first user interface object optionally providesvisual guidance about the requirements for completing the selectioninput. In some embodiments, user interactions with the representation ofthe first user interface object is translated into interaction with thefirst user interface object, and causes the computer system to performoperations with respect to the first user interface object in accordancewith the interaction between the first user 7102 and the representationof the first user interface object. In some embodiments, therepresentation of the first user interface object remains displayed atthe position of the representation 7202′ of the first user's hand 7202to indicate that the first user 7102 has control of the first userinterface object, optionally, in exclusion of interaction of other usersthat are sharing the three-dimensional environment with the first user.

In some embodiments, some or all the features described above withrespect to the behaviors of the computer systems, the first displaygeneration component 7200 and the second display generation component7100 in FIGS. 7A-7C are equally applicable to other scenarios where theroles of the first user 7102 and the second user 7002 with respect tothe first user interface object 7016 are reversed. In such otherscenarios, the operations of the computer systems and display generationcomponents used by the first user and the second user may be reversedaccordingly in a particular scenario. The features described above arestill valid, and therefore not repeated herein in the interest ofbrevity.

FIGS. 7D-7F are block diagrams illustrating a method of displaying arepresentation of a physical object relative to a viewpoint of acurrently displayed view of a three-dimensional environment in differentmanners, where the viewpoint moves in accordance with movement of theuser in a first physical environment, the representation of the physicalobject moves in accordance with movement of the physical object in asecond physical environment different from the first physicalenvironment, and where a change in the manner of displaying therepresentation is triggered in response to a spatial relationshipbetween the representation of the physical object and the viewpointmeeting preset criteria, in accordance with some embodiments.

In some embodiments, the computer system displays a view of athree-dimensional environment 7304 that includes a representation of aphysical object (e.g., a second user 7102, an animal, a moving drone,etc.) that is located in a different physical environment (e.g., scene105-b, or another indoor or outdoor physical environment, etc.) from thephysical environment (e.g., scene 105-a, or another indoor or outdoorphysical environment, etc.) of a first user (and of a first displaygeneration component 7100 used by the first user 7002 to view thethree-dimensional environment 7204). The computer system, optionally,moves the viewpoint corresponding to the currently displayed view of thethree-dimensional environment 7304 in accordance with the movement ofthe first user 7002 (and/or the first display generation component 7100)in their physical environment (e.g., scene 105-a, or another physicalenvironment, etc.). The computer system determines the position andmovement path of the representation of the physical object (e.g.,representation 7102′-a of the second user 7102, representation ofanother physical object, etc.) in the three-dimensional environment 7204based on a location and movement path of the physical object in itsphysical environment (e.g., scene 105-b, or another physicalenvironment, etc.). The computer system utilizes a first type ofcorrespondence (e.g., mapping and conversion relationships; optionally,different mapping and conversion relationships for the viewpoint, thephysical object, and the first user, etc.) between positions in thethree-dimensional environment 7304 and locations in a respectivephysical environment (e.g., the physical environment 105-a of the firstuser 7002 and the first display generation component 7100, the physicalenvironment of the physical object (e.g., physical environment 105-b ofthe second user 7102, another physical environment of the physicalobject, etc.), etc.). Under some conditions (e.g., due to movement ofthe first user 7002, and/or movement of the physical object (e.g., aphysical object represented by the second user 7102 in this example),etc.), the position of the representation of the physical object wouldbe within a threshold distance (e.g., an arm's length, three feet, auser-specified distance, etc.) of the position of the viewpoint of thecurrently displayed view (e.g., view 7304-a, 7304-a′, etc.) of thethree-dimensional environment 7304 shown via the first displaygeneration component 7100, if the position(s) are determined using thefirst type of correspondence between positions in the three-dimensionalenvironment 7304 and locations in the physical environments (e.g.,scenes 105-a, 105-b, etc.). Under such conditions, the computer systemdisplays the representation of the physical object (e.g., representation7102′-a, in this example) at an adjusted position that is offset fromthe position determined based on the first type of correspondence (e.g.,as shown in FIG. 7F). In some embodiments, the adjusted position isdetermined based on a second type of correspondence that is differentfrom the first type of correspondence and ensures that the adjustedposition remains more than the threshold distance from the position ofthe viewpoint of the currently displayed view of the three-dimensionalenvironment shown via the first display generation component (e.g., view7304-a″, subsequent views shown via the first display generationcomponent 7100, etc.). The computer system continues to use the secondtype of correspondence to determine the adjusted position of therepresentation of the physical object (e.g., representation 7102′-a, inthis example), until the unadjusted position calculated based on thefirst type of correspondence is more than the threshold distance awayfrom the position of the viewpoint of the currently displayed view ofthe three-dimensional environment shown via the first display generationcomponent (e.g., view 7304-a″, subsequent views shown via the firstdisplay generation component 7100, etc.).

In some embodiments, when a computer system provides a view of athree-dimensional environment 7304 to a first user 7002, and theposition of the viewpoint corresponding to the currently displayed viewof the three-dimensional environment 7304 is based on the location ofthe first user's head, body, or eyes, in the physical environment of thefirst user 7002, the computer system sometimes displays representationsof other physical objects (e.g., a physical object represented by thesecond user 7102 in this example, but may be an inanimate object or ananimate object that is not sharing the computer-generated environment7304 with the first user 7002, etc.) at positions corresponding tolocations of the physical objects in their respective physicalenvironment. In some circumstances, even though there is no danger orpossibility of actual physical collision or uncomfortable spatialproximity between the first user 7002 and the other physical objects inthe real world, the positions of the representations of the physicalobjects may collide with or get too close to the position of theviewpoint corresponding to the view shown to the first user (e.g., ifnot specifically adjusted, otherwise addressed, etc.), and making thevisual experience of the first user in the three-dimensional environmentuncomfortable or jarring to the first user at times.

As disclosed herein, the computer system determines the position for arepresentation of a physical object located in a different physicalenvironment from the first user based on a first type of correspondenceor mapping relationship between positions in the three-dimensionalenvironment and corresponding locations in a physical environment thephysical object, when the position of the representation of the physicalobject determined based on the first type of correspondence is notwithin a threshold range of the viewpoint corresponding to the currentlydisplayed view of the three-dimensional environment shown to the firstuser. That means, if the representation of the physical object is at adistance from the virtual position of the viewpoint, the movement of therepresentation of the physical object in the three-dimensionalenvironment can correspond to the movement of the physical object in amanner that mimics movement and spatial relationships in the real worldand the representation of the physical object would not invade the senseof personal space of the first user. However, if the representation ofthe physical object is very close from the virtual position of theviewpoint, the movement of the representation of the physical objectthat correspond to the movement of the physical object in the samemanner (e.g., accordance with the first type of correspondence ormapping relationship) would cause the representation of the physicalobject to be displayed with an unreasonable size, overlap with theviewpoint, and/or invade the sense of personal space of the first user.Accordingly, in accordance with a determination that the representationof the physical object would be within a threshold distance from theviewpoint based on the first type of correspondence or mappingrelationship, the computer system uses a second type of correspondenceor mapping relationship between positions in the three-dimensionalenvironment and corresponding locations in the physical environment ofthe physical object to calculate an adjusted position for therepresentation of the physical object, such that the representation ofthe physical object can be displayed at the adjusted position and/ormove in a manner to avoid being displayed with an unreasonable size,overlapping with the viewpoint, and/or invading the sense of personalspace of the first user.

FIG. 7D illustrates a scenario in which two users, e.g., the first user7002 and the second user 7102 are sharing a computer-generatedthree-dimensional environment 7304, in accordance with some embodiments.In some embodiments, the first user 7002 is located in a first physicalenvironment 105-a, and the second user 7102 is located in a secondphysical environment 105-b. In some embodiments, the first physicalenvironment and the second physical environment are parts of the samephysical environment that may overlap with each other. In someembodiments, the first physical environment and the second physicalenvironment are separate physical environments that do not overlap witheach other. In some embodiments, the first physical environment and thesecond physical environment are optionally indoor environments, outdoorenvironments, one indoor and one outdoor environment, a mix of indoorand outdoor environments, etc. In this example, the first physicalenvironment includes physical surfaces (e.g., walls 7004-a and 7006-a,floor 7008-a, etc.) and physical objects (e.g., physical object 7010,other physical objects, etc.); and the second physical environmentincludes physical surfaces (e.g., walls 7004-b and 7006-b, floor 7008-b,etc.) and physical objects (e.g., physical object 7014, other physicalobjects, etc.). The first user 7002 is a user of the first displaygeneration component 7100 and is provided with a first view 7304-a (andsubsequently updated first views 7304-a′, 7304-a″, etc.) of the sharedthree-dimensional environment 7304 via the first display generationcomponent 7100. The second user 7102 is a user of the second displaygeneration component 7200 and is provided with a second view 7304-b (andsubsequently updated first views 7304-b′, 7304-b″, etc.) of the sharedthree-dimensional environment 7304 via the second display generationcomponent 7200. For illustrative purposes, the first user 7002 movesforward along a straight line 7300 in the first physical environment105-a, and the second user 7102 moves forward along a straight line 7302in the second physical environment 105-b, where the representation 7300′of the straight line 7300 in the second view 7304-b of thethree-dimensional environment 7304 passes through the viewpoint of thesecond view 7304-b; and the representation 7302′ of the straight line7302 in the first view 7304-a of the three-dimensional environment 7304passes through the viewpoint of the first view 7304-a. In someembodiments, there is no requirement that the movement paths of thefirst user 7002 and the second user 7102 should be straight lines, andthe paths may be in any shapes and/or have any spatial extents suitablein their physical environments. In some embodiments, there is norequirement that the first user and the second user both move in theirrespective physical environment. In some embodiments, the viewpoint ofthe currently displayed view of the three-dimensional environmentprovided via a respective display generation component may not bestationary, and/or may move in accordance with the movement of therespective display generation component and/or the movement of therespective user of the respective display generation component. In someembodiments, there is no requirement that the three-dimensionalenvironment is a shared environment between the first user and thesecond user. For example, in some embodiments, from the perspective ofthe first display generation component 7100, the second user 7102 inthis example is merely a representation of a physical object (e.g., ananimal, a drone, a person that is not using or providing input to thethree-dimensional environment, etc.) in the second physical environment.Similarly, in some embodiments, from the perspective of the seconddisplay generation component, the first user 7002 in this example ismerely a representation of a physical object (e.g., an animal, a drone,a person that is not using or providing input to the three-dimensionalenvironment, etc.) in the first physical environment. In someembodiments, only one of the display generation components (e.g., thefirst display generation component, the second display generationcomponent, etc.) is used, and the other display generation componentdoes not exist or participate in the processes described herein.

In the example shown in FIG. 7D, the three-dimensional environment 7304is shared between the user 7002 and the user 7102 in response to arequest that is initiated from one of the users 7002 and 7102 using acomputer system controlled by said one user, and accepted by another ofthe users 7002 and 7102 using a computer system controlled by saidanother user, in accordance with some embodiments. In some embodiments,both users have received and accepted the request to share thethree-dimensional environment using their respective computer systemsfrom the computer system used by a third user. In some embodiments, bothusers have sent requests to share the three-dimensional environment to aserver using their respective computer systems, where their requestswere accepted by the server. When sharing the computer-generatedthree-dimensional environment, the locations and orientations of theusers and of their respective heads, eyes, hands, arms, and/or wristsare captured in real-time or periodically by sensors (e.g., cameras,motion sensors, etc.) and the location and orientation data is providedto one or both of the computer systems controlled by the users, and/orto a server that is in communication with the computer systems. Thelocation data is used by the computer systems and/or server to determinethe respective locations and orientations of the users and of theirrespective heads, eyes, hands, arms, and/or wrists in thecomputer-generated three-dimensional environment, and correspondingly,the respective positions of the representations of the users includingtheir respective heads, arms, hands, and/or wrists in the views of thethree-dimensional environment provided via the different displaygeneration components associated with the users, as well as the viewingperspectives of the views of the three-dimensional environment providedvia the different display generation components associated with theusers. In some embodiments, the computer-generated environment shared bythe users is an environment of a virtual conference call, a chatsession, a multi-player game, a shared computer-generated experience(e.g., group meditation, exercise, game, collaborative work, etc.), etc.In some embodiments, the representation of the users are respectiveavatars of the users. In some embodiments, the representations of theusers optionally are not attached to or supported by a surface in thethree-dimensional environment.

In FIG. 7D, part (A), the computer system displays the first view 7304-aof the three-dimensional environment 7304 via the first displaygeneration component 7100. In the first view 7304-a of thethree-dimensional environment, a representation 7102′-a of the seconduser 7102 is displayed at a position that corresponds to the currentlocation of the second user 7102 in the second physical environment105-b. There are other objects in the first view 7304-a of thethree-dimensional environment, such as a virtual path 7306-a, a virtualobject 7308-a, etc. The respective appearances and display positions ofthe representation 7102′-a of the second user 7102, the virtual object7308-a, and the virtual path 7306-a in the first view 7304-a are basedon their respective positions in the three-dimensional environmentrelative to the position of the viewpoint of the currently displayedfirst view 7304-a of the three-dimensional environment shown via thefirst display generation component 7100. In some embodiments, arepresentation 7002′-a of the first user 7002 is, optionally, visible inthe first view 7304-a of the three-dimensional environment, at aposition that corresponds to the virtual position of the first user 7002and/or the viewpoint of the currently displayed first view 7304-a in thethree-dimensional environment. In this example, as shown in FIG. 7D,part (A), the computer system displays movement of the representation7102′-a along the representation 7302′ of the straight line 7302 towardthe virtual position of the viewpoint of the first view 7304-a. In themoment depicted in FIG. 7D, the representation 7102′-a is displayed at aposition that is calculated in accordance with the first type ofcorrespondence between the positions in the three-dimensionalenvironment 7304 and the locations in the second physical environment(e.g., scene 105-b, or another physical environment of the second user7102, etc.). The representation 7102′-a of the second user 7102 is shownto move toward and approach the viewpoint of the first view 7304-a asthe second user 7102 moves forward along the line 7302 in the secondphysical environment.

In some embodiments, as shown in FIG. 7D, part (B), the computer systemor another computer system that is in communication with the computersystem, optionally, displays the second view 7304-b of thethree-dimensional environment 7304 via the display generation component7200. In the second view 7304-b of the three-dimensional environment, arepresentation 7002′-b of the first user 7002 is displayed at a positionthat corresponds to the current location of the first user 7002 in thefirst physical environment (e.g., scene 105-a, or another physicalenvironment of the first user, etc.). There are other objects in thesecond view 7304-b of the three-dimensional environment, such as avirtual path 7306-b (e.g., same virtual path as the virtual path 7306-abut viewed from the viewpoint of the second view 7304-b), a virtualobject 7308-b (e.g., the same virtual object as the virtual object7308-a but viewed from the viewpoint of the second view 7304-b), etc.The respective appearances and display positions of the representation7002′-b of the first user 7002, the virtual object 7308-b, and thevirtual path 7306-b in the second view 7304-b are based on theirrespective positions in the three-dimensional environment relative tothe position of the viewpoint of the currently displayed second view7304-b of the three-dimensional environment shown via the second displaygeneration component 7200. In some embodiments, a representation 7102′-bof the second user 7102 is visible in the second view 7304-b of thethree-dimensional environment, at a position that corresponds to thevirtual position of the second user 7102 and/or the viewpoint of thecurrently displayed second view 7304-b. In this example, as shown inFIG. 7D, part (B), the computer system displays movement of therepresentation 7002′-b along the representation 7300′ of the straightline 7300 toward the virtual position of the viewpoint of the secondview 7304-b. In the moment depicted in FIG. 7D, the representation7002′-b is displayed at a position that is calculated in accordance withthe first type of correspondence between the positions in thethree-dimensional environment 7304 and the locations in the firstphysical environment (e.g., the scene 105-a, or another physicalenvironment of the first user, etc.). The representation 7002′-b of thefirst user 7002 is shown to move toward and approach the virtualposition of the viewpoint of the second view 7304-b as the first user7002 moves forward along the line 7300 in the first physicalenvironment.

FIG. 7E illustrates a point in time where either or both the first user7002 and the second user 7102 have moved in their respective physicalenvironments such that the respective positions of the first user andthe second user in the three-dimensional environment 7304 as calculatedin accordance with the first type of correspondence (e.g., the firsttype of correspondence between the positions in the three-dimensionalenvironment 7304 and the locations in the first physical environment,the first type of correspondence between the positions in thethree-dimensional environment 7304 and the locations in the secondphysical environment, etc.) are at a respective preset thresholddistance of each other in the three-dimensional environment 7304. Insome embodiments, at this time, as illustrated in FIG. 7E, part (A), therespective position of the representation 7102′-a of the second user7102 and the position of the viewpoint of the updated first view 7304-a′as calculated in accordance with the first type of correspondencebetween the positions in the three-dimensional environment 7304 and thelocations in the second physical environment is at a first thresholddistance of each other in the three-dimensional environment 7304. Insome embodiments, optionally, as illustrated in FIG. 7E, part (B), therespective position of the representation 7002′-b of the first user 7002and the position of the viewpoint of the updated second view 7304-b′ ascalculated in accordance with the first type of correspondence betweenthe positions in the three-dimensional environment 7304 and thelocations in the first physical environment is at a second presetthreshold distance (e.g., the same as the first preset thresholddistance, at a different preset threshold distance, etc.) of each otherin the three-dimensional environment. In some embodiments, the firstthreshold distance is different from the second threshold distance,depending on the respective personal settings and other characteristics(e.g., size, shape, posture, activity, etc.) of the first user and thesecond user.

In FIG. 7E, part (A), in response to detecting the movement of the firstuser 7002 and the second user 7102 in his/her physical environment, thecomputer system displays an updated first view 7304-a′ of thethree-dimensional environment with a viewpoint that is moved inaccordance with the movement of the first user 7002 in the firstphysical environment. In some embodiments, the viewpoint of the updatedfirst view 7304-a′ is stationary in the three-dimensional environment ifthe first user 7002 and/or the first display generation component 7100did not move in the first physical environment. In some embodiments, inaccordance with a determination that the respective position of therepresentation 7102′-a of the second user 7102 in the three-dimensionalenvironment that is calculated based on the current location of thesecond user in the second physical environment in accordance with thefirst type of correspondence is more than or not less than the firstpreset threshold distance from a respective position in thethree-dimensional environment that corresponds to the viewpointassociated with the updated first view 7304-a′ of the three-dimensionalenvironment, the computer system displays the representation 7102′-a ata first display position in the updated first view 7304-a′ of thethree-dimensional environment, where the first display position is therespective position of the representation 7102′-a in thethree-dimensional environment.

In some embodiments, the first preset threshold distance is an arm'slength, a preset radius of a personal space for the first user 7002 inthe three-dimensional environment 7304, defined by a preset boundarysurface surrounding a virtual position of the first user 7002 in thethree-dimensional environment (e.g., the virtual surface of therepresentation of the first user 7002, or a bounding box surrounding thevirtual position of the first user 7002).

In some embodiments, optionally, as shown in FIG. 7E, part (B), inresponse to detecting the movement of the second user 7102 in his/herphysical environment, the computer system of the second user 7102displays an updated second view 7304-b′ of the three-dimensionalenvironment 7304 with a viewpoint that is moved in accordance with themovement of the second user 7102 in the second physical environment. Insome embodiments, the viewpoint of the updated second view 7304-b′ isstationary in the three-dimensional environment if the second user 7102and/or the second display generation component 7200 did not move in thesecond physical environment. In some embodiments, in accordance with adetermination that the respective position of the representation 7002′-bof the first user 7002 in the three-dimensional environment that iscalculated based on the current location of the first user 7002 in thefirst physical environment in accordance with the first type ofcorrespondence is more than or not less than the second preset thresholddistance from a respective position in the three-dimensional environmentthat corresponds to the viewpoint associated with the updated secondview 7304-b′ of the three-dimensional environment, the computer systemof the second user 7102 displays the representation 7002′-b at a seconddisplay position in the updated second view 7304-b′ of thethree-dimensional environment, where the second display position is therespective position of the representation 7002′-b in thethree-dimensional environment.

In some embodiments, the second preset threshold distance is an arm'slength, a preset radius of a personal space for the second user 7102 inthe three-dimensional environment, defined by a preset boundary surfacesurrounding a virtual position of the second user 7102 in thethree-dimensional environment (e.g., the virtual surface of therepresentation of the second user 7102, a bounding box surrounding thevirtual position of the second user 7102, etc.), etc.

In FIG. 7F, the next moment after that shown in FIG. 7E, as the movementof either or both the first user 7002 and the second user 7102 continuedin their respective physical environments such that the respectivepositions of the first user and the second user in the three-dimensionalenvironment as calculated in accordance with the first type ofcorrespondence (e.g., the first type of correspondence between thepositions in the three-dimensional environment 7304 and the locations inthe first physical environment, the first type of correspondence betweenthe positions in the three-dimensional environment 7304 and thelocations in the second physical environment, etc.) are within therespective preset threshold distance of each other in thethree-dimensional environment. In some embodiments, at this time, therespective position of the representation of the second user 7102 andthe position of the viewpoint of the further updated first view 7304-a″as calculated in accordance with the first type of correspondencebetween the positions in the three-dimensional environment 7304 and thelocations in the second physical environment is less than the firstpreset threshold distance of each other in the three-dimensionalenvironment.

In FIG. 7F, part (A), in response to detecting the further movement ofthe first user 7002 and/or the second user 7102 in their respectivephysical environments, the computer system displays the further updatedfirst view 7304-a″ of the three-dimensional environment with a viewpointthat is moved in accordance with the further movement of the first user7002 in the first physical environment. In some embodiments, theviewpoint of the further updated first view 7304-a″ is stationary in thethree-dimensional environment if the first user 7002 and/or the firstdisplay generation component 7100 did not move in the first physicalenvironment. In some embodiments, in accordance with a determinationthat the respective position of the representation 7102′-a of the seconduser 7102 in the three-dimensional environment that is calculated basedon the current location of the second user 7102 in the second physicalenvironment in accordance with the first type of correspondence is lessthan the first preset threshold distance from a respective position inthe three-dimensional environment that corresponds to the viewpointassociated with the further updated first view 7304-a″ of thethree-dimensional environment, the computer system displays therepresentation 7102′-a at an adjusted display position in the furtherupdated first view 7304-a″ of the three-dimensional environment, wherethe adjusted display position is offset from the respective position ofthe representation 7102′-a in the three-dimensional environment at thismoment. For example, in FIG. 7F, part (A), instead of displaying therepresentation 7102′-a at a position that is straight in front of therepresentation 7002′-a or overlapping with the representation 7002′-a inthe further updated first view 7304-a″, the adjusted display position ofthe representation 7002′-a is offset to the side (e.g., right side, oranother side or direction, etc.) of the representation 7002′-a of thefirst user 7002. In general, instead of displaying the representation7102′-a at a position that is within the first preset threshold distanceof the viewpoint of the currently displayed first view 7304-a″, thecomputer system displays the representation 7102′-a at an adjusteddisplay position that is offset from the unadjusted position calculatedin accordance with the first type of correspondence. In someembodiments, the computer system continues to apply the adjustment tothe display position of the representation 7102′-a during the movementof the first user 7002 and/or the second user 7102, until the distancebetween the position of the representation 7102′-a and the position ofthe viewpoint of the currently displayed first view 7304-a″ are nolonger within the first preset threshold distance of each other.

In some embodiments, optionally, as illustrated in FIG. 7F, part (B),the respective position of the representation 7002′-b of the first user7002 and the position of the viewpoint of the further updated secondview 7304-b″ as calculated in accordance with the first type ofcorrespondence between the positions in the three-dimensionalenvironment 7304 and the locations in the first physical environment isless than the second preset threshold distance (e.g., the same as thefirst preset threshold distance, at a different preset thresholddistance, etc.) of each other in the three-dimensional environment.

In FIG. 7F, part (B), in response to detecting the further movement ofthe first user 7002 and/or the second user 7102 in their respectivephysical environments, the computer system of the second user 7102displays a further updated second view 7304-b″ of the three-dimensionalenvironment with a viewpoint that is moved in accordance with thefurther movement of the second user 7102 in the second physicalenvironment. In some embodiments, the viewpoint of the further updatedsecond view 7304-a″ is stationary in the three-dimensional environmentif the second user 7102 and/or the second display generation component7200 did not move in the second physical environment. In someembodiments, in accordance with a determination that the respectiveposition of the representation 7002′-b of the first user 7002 in thethree-dimensional environment that is calculated based on the currentlocation of the first user 7002 in the first physical environment inaccordance with the first type of correspondence is less than the secondpreset threshold distance from a respective position in thethree-dimensional environment that corresponds to the viewpointassociated with the further updated second view 7304-b″ of thethree-dimensional environment, the computer system of the second user7102 displays the representation 7002′-b at an adjusted display positionin the further updated second view 7304-b″ of the three-dimensionalenvironment, where the adjusted display position is offset from therespective position of the representation 7002′-b in thethree-dimensional environment at this moment. For example, in FIG. 7F,part (B), instead of displaying the representation 7002′-b at a positionthat is straight in front of the representation 7102′-b or overlappingwith the representation 7102′-b in the further updated second view7304-b″, the adjusted display position of the representation 7002′-b isoffset to the side (e.g., to the right, to another side or direction,etc.) of the representation 7102′-b of the second user 7102. In general,instead of displaying the representation 7002′-b at a position that iswithin the second preset threshold distance of the viewpoint of thecurrently displayed second view 7304-b″, the computer system of thesecond user 7102 displays the representation 7002′-b at an adjusteddisplay position that is offset from the unadjusted position calculatedin accordance with the first type of correspondence. In someembodiments, the computer system of the second user 7102 continues toapply the adjustment during the movement of the first user and/or thesecond user, until the distance between the position of therepresentation 7002′-b and the position of the viewpoint of thecurrently displayed second view 7304-b″ are no longer within the presetsecond threshold distance of each other.

In some embodiments, in the above example, the first user 7002 ismoving, and the second user 7102 is stationary. As a result, unlessadjusted in the manner described above, the viewpoint of the currentlydisplayed view 7304-a, 7304-a′, and 7304-a″ have different positions inthe three-dimensional environment; and the representations 7002′-b ofthe first user 7002 has the different positions in the three-dimensionalenvironment (e.g., in the currently displayed first view 7304-a,7304-a′, and 7304-a″ and the currently displayed second view 7304-b,7304-b′, 7304-b″ in FIGS. 7D-7F). Unless as adjusted in the mannerdescribed above, the viewpoint of the currently displayed view 7304-b,7304-b′, and 7304-b″ have the same position in the three-dimensionalenvironment; and the representations 7102′-a of the second user 7102 hasthe same position in the three-dimensional environment (e.g., in thecurrently displayed first view 7304-a, 7304-a′, and 7304-a″ and thecurrently displayed second view 7304-b, 7304-b′, 7304-b″ in FIGS.7D-7F).

In some embodiments, in the above example, the first user 7002 isstationary, and the second user 7102 is moving in the second physicalenvironment. As a result, unless adjusted in the manner described above,the viewpoint of the currently displayed view 7304-b, 7304-b′, and7304-b″ have different positions in the three-dimensional environment;and the representations 7102′-a of the second user 7102 has thedifferent positions in the three-dimensional environment (e.g., in thecurrently displayed first view 7304-a, 7304-a′, and 7304-a″ and thecurrently displayed second view 7304-b, 7304-b′, 7304-b″ in FIGS.7D-7F). Unless as adjusted in the manner described above, the viewpointof the currently displayed view 7304-a, 7304-a′, and 7304-a″ have thesame position in the three-dimensional environment; and therepresentations 7002′-b of the first user 7002 has the same position inthe three-dimensional environment (e.g., in the currently displayedfirst view 7304-a, 7304-a′, and 7304-a″ and the currently displayedsecond view 7304-b, 7304-b′, 7304-b″ in FIGS. 7D-7F).

In some embodiments, in the above example, the first user 7002 and thesecond user 7102 are both moving in their respective physicalenvironments. As a result, the viewpoints of the currently displayedfirst view 7304-b, 7304-b′, and 7304-b″, and the viewpoints of thecurrently displayed second view 7304-a, 7304-a′, 7304-a″, all havedifferent positions in the three-dimensional environment; therepresentations 7102′-a of the second user 7102 has the differentpositions in the three-dimensional environment in the currentlydisplayed first view 7304-a, 7304-a′, and 7304-a″ and the currentlydisplayed second view 7304-b, 7304-b′, and 7304-b″ in FIGS. 7D-7F; andthe representations 7002′-b of the first user 7002 has the differentpositions in the three-dimensional environment in the currentlydisplayed first view 7304-a, 7304-a′, and 7304-a″ and the currentlydisplayed second view 7304-b, 7304-b′, and 7304-b″ in FIGS. 7D-7F.

In some embodiments, the representation 7002′-b of the first user 7002and/or the representation 7102′-a of the second user 7102 are floatingin space in the first view and the second view. For example, in someembodiments, the representation 7002′-b of the first user 7002 is afloating avatar of the first user 7002 that floats in the second view7034-b, 7034-b′, and 7034-b″, etc. of the three-dimensional environment,and automatically moves out of the way as the viewpoint of the secondview 7034-b″ gets within the second preset threshold distance of therepresentation 7002′-b, due to movement of the first user and/or themovement of the second user. Similarly, in some embodiments, therepresentation 7102′-a of the second user 7102 is a floating avatar ofthe second user 7102 that floats in the first view 7034-a, 7034-a′, and7034-a″, etc. of the three-dimensional environment, and automaticallymoves out of the way as the viewpoint of the first view 7034-a″ getswithin the first preset threshold distance of the representation7102′-a, due to movement of the first user and/or the movement of thesecond user. In some embodiments, the avatars of the users in thethree-dimensional environment have a level of realism that is selectedbased on the level of realism of the three-dimensional environment(e.g., photographic level of realism, cartoon level of realism, etc.).In some embodiments, in accordance with a determination that thethree-dimensional environment 7304 is displayed with a first level ofrealism, the representations of the users are displayed with a first setof display properties (e.g., first resolution, first number ofdimensions, first level of clarity, first color palette, withoutlighting effect, etc.) that corresponds to the first level of realism,and in accordance with a determination that the three-dimensionalenvironment is displayed with a second level of realism that isdifferent from (e.g., greater than, less than, etc.) the first level ofrealism, the representations of the users is displayed with a second setof display properties (e.g., second resolution, second number ofdimensions, second level of clarity, second color palette, with lightingeffect, etc.) that corresponds to the second level of realism, thesecond set of display properties are different from (e.g., greater than,less than, adding to, subtracting from, etc.) the first set of displayproperties.

In some embodiments, when the display position of the representation ofa respective user is adjusted, the representation of the respective usermoves with a movement component that does not correspond to movement ofthe respective user in the physical environment in the usual manner(e.g., in accordance with the first type of correspondence, withoutadjustment, etc.). In some embodiments, the amount of offset that isapplied to the adjusted position of the respective representation of arespective user is variable based on the spatial relationship betweenthe respective representation and the virtual position of the viewpointin the three-dimensional environment. In some embodiments, theadjustment to the display position of the representation 7102′-a isoptionally applied to the first view 7304-a″ displayed to the first user7002, and not to the second view 7304-b″ displayed to the second user7102. In some embodiments, the adjustment to the display position of therepresentation 7002′-b is optionally applied to the second view 7304-b″displayed to the second user 7102, and not to the first view 7304-a″displayed to the first user 7002.

In some embodiments, the three-dimensional environment 7304 includes avirtual three-dimensional environment or an augmented realityenvironment, and the first user and the second user have a sharedexperience in the virtual three-dimensional environment. In someembodiments, the positions and movements of the first user and thesecond user in their respective physical environments (e.g., samephysical environment, different physical environments, etc.) are mapped(e.g., using the same mapping relationship, or different mappingrelationship, etc.) to positions and movements in the samethree-dimensional environment, but the appearance of thethree-dimensional environments may be adjusted (e.g., with differentwallpapers, color schemes, with different virtual furniture, etc.) totailor to a respective user in the view of the three-dimensionalenvironment shown to the respective user.

FIGS. 7G-7J are block diagrams illustrating changing a level ofimmersion with which an environment of a computer-generated experienceis displayed in accordance with changing biometric data of a user thatis received by the computer system, in accordance with some embodiments.

In some embodiments, the computer system changes the level of immersionwith which a computer-generated experience (e.g., visual experience,audio-visual experience, virtual reality experience, augmented realityexperience, etc.) is presented to a user in accordance with biometricdata (e.g., biometric data represented by bar 7312, other biometricdata, etc.) corresponding to the user (e.g., user 7002). For example,when the user is adjusting his/her physical and emotional states afterthe computer-generated experience is started, e.g., proactively and/orunder the influence of the computer-generated content, the computersystem may detect changes in the biometric data (e.g., heart rate, bloodpressure, breathing rate, etc.) corresponding to the user. In accordancewith the changes in the biometric data relative to respective sets ofpreset criteria associated with different levels of immersion (e.g., athreshold represented by indicator 7326, or other types of thresholds orcriteria, etc.), the computer system increases or decreases the level ofimmersion with which the computer-generated experience is provided tothe user (e.g., by changing the visual prominence (e.g., includingspatial extent, visual depth, color saturation, visual contrast, etc.)of virtual content relative to the visual prominence of therepresentation of the physical environment (e.g., by enhancingcomplexity, spatial extent, and/or visual characteristics of the virtualcontent, and/or reducing the visual clarity, blur radius, opacity, colorsaturation, etc. of the representation of the physical environment,etc.).

In the example shown in FIG. 7G-7J, the computer system initiallydisplays a view 7316 of a three-dimensional environment via a displaygeneration component (e.g., display generation component 7100, oranother type of display generation component such as an HMD, etc.). Insome embodiments, the view 7316 of the three-dimensional environment isa pass-through view of a physical environment of the user 7002, and doesnot include virtual content or includes a minimal amount of virtualcontent (e.g., system controls, indicators, etc.) in peripheral portionsof the field of view provided by the display generation component. Theview 7316 corresponds to a low level of immersion with which acomputer-generated experience is provided to a user, e.g., due to theminimal amount of virtual content that is displayed relative to therepresentation of the user's physical environment. In this example, theview 7316 of the three-dimensional environment includes representationsof physical surfaces (e.g., representations 7004′ and 7006′ of twoadjacent walls 7004 and 7006, a representation 7008′ of a floor 7008,etc. in the physical environment 105 of the user 7002), andrepresentations of physical objects (e.g., a representation 7010′ of aphysical object 7010, and representations of other physical objects,etc. in the physical environment 105 of the user 7002).

FIG. 7G also illustrates that, the computer system, while displaying theview 7316 of the three-dimensional environment with the low level ofimmersion (e.g., displaying a pass-through view of the physicalenvironment, or displaying a representation of the physical environmentwith a minimal amount of virtual content, etc.), the computer systemreceives biometric data corresponding to the user 7002. In accordancewith a determination that the biometric data of the user 7002 does notmeet preset criteria corresponding to a next higher level of immersion,the computer system maintains display of the first view 7316 of thethree-dimensional environment, without reducing visual prominence of therepresentation of the physical environment in the currently displayedview of the three-dimensional environment. For example, as illustratedin FIG. 7G, the biometric data has a value or set of values indicated bythe length of the bar 7312 relative to a full range of value(s) for thebiometric data, and threshold values corresponding to the presetcriteria to transition into a different, higher level of immersion isindicated by the position of the indicator 7326 relative to the fullrange of values for the biometric data.

In some embodiments, the biometric data corresponding to the user 7002include one or more of a heart rate, a breathing rate, a bodytemperature, a serum concentration of certain chemicals, medication,and/or hormones, etc., a blood pressure, brain waves, a focus level, apupil size, a metabolic rate, a blood sugar level, etc., of the user7002. In some embodiments, the biometric data corresponding the user7002 include one or more types of biometric data (e.g., breathing rate,blood pressure, focus level, blood sugar level, etc.) that may vary overtime during a user's engagement with the computer-generated experience.In some embodiments, the biometric data corresponding to the userinclude one or more types of biometric data that may vary through theuser's physical actions (e.g., meditation, breathing pattern change,exercise, etc., as opposed to direct interaction with user interfaceelements or controls provided by the computer system during the user'sengagement with the computer-generated experience). In some embodiments,the biometric data corresponding to the user includes one or more typesof composite metrics of multiple types of biometric data that correspondto a user's mood, happiness, and/or stress level, etc. In someembodiments, the biometric data include real-time data that correspondto the physiological state of the user at the time or within a presetamount of time prior to the display of the current view of thethree-dimensional environment via the display generation component. Insome embodiments, the biometric data is collected continuously and/orperiodically through one or more biometric sensors (e.g., varioussuitable medical devices, vibration sensors, cameras, thermal sensors,chemical sensors, etc.) connected to or pointed at the user, andcontinuously and/or periodically transmitted to the computer system. Insome embodiments, the biometric data does not include non-transientcharacteristics of humans (e.g., fingerprint, iris pattern and color,facial features, voiceprint, etc.) that do not typically change over aperiod of time that an average user is engaged with thecomputer-generated experience.

In some embodiments, the computer system determines that the biometricdata does not meet the preset criteria for transitioning to displayingthe computer-generated experience with a preset higher level ofimmersion in accordance with a determination that the heart rate isgreater than a first threshold heart rate, the blood pressure is higherthan a first threshold blood pressure, the movement of the user is morethan a first threshold amount of movement during a threshold amount oftime, the body temperature of the user is higher than a first thresholdbody temperature, the metric of stress level is above a first thresholdstress level, the metric corresponding to the user's mood indicates thatthe user is agitated and unhappy, etc. In some embodiments, the computersystem directly switches to displaying the three-dimensional environmentwith the preset higher level of immersion (e.g., as shown in FIG. 7J)when the preset criteria are met, without going through gradualtransitions based on changes in the biometric data before the presetcriteria are met. In some embodiments, optionally, thecomputer-generated experience includes visual and/or audio guidance(e.g., music, scenery, inspirational messages, guided medicationrecording, visual, audio, or verbal instructions on breathing, etc.)helping the user to enter into a state in which the correspondingbiometric data received from the user will meet the preset criteria.

FIGS. 7H-7I illustrate that, in some embodiments, the computer systemgradually adjust the level of immersion with which thecomputer-generated experience is provided to the user in accordance withthe trend and/or magnitude of changes in the biometric datacorresponding to the user. For example, in some embodiments, with thebiometric data exhibits a change approaching satisfaction of the presetcriteria for switching to the preset higher level of immersion (e.g., anaugmented reality view, an augmented virtuality view, a virtual realityview, etc.), the computer system increase the visual prominence and/oramount of virtual content corresponding to the computer-generatedexperience, and reduces the visual prominence and/or amount of therepresentation of the physical environment in the currently displayedview of the three-dimensional environment. In some embodiments, thecomputer system changes the visual balance between the virtual contentcorresponding to the computer-generated experience and therepresentation of the physical environment by an amount that correspondsto the amount and/or nature of the change in the biometric datacorresponding to the user. Similarly, in some embodiments, with thebiometric data exhibiting a change away from satisfaction of the presetcriteria for switching to the preset higher level of immersion, thecomputer system decreases the visual prominence and/or amount virtualcontent corresponding to the computer-generated experience and increasesthe visual prominence and/or amount of the representation of thephysical environment in the currently displayed view of thethree-dimensional environment.

In some embodiments, the computer system changes the visual balancebetween the virtual content and the representation of the physicalenvironment by an amount that corresponds to the amount and/or nature ofthe change in the biometric data corresponding to the user. As shown inFIG. 7H, when the values of the biometric data change toward meeting thepreset criteria (e.g., as indicated by the increased length of bar 7312approaching the position of the indicator 7326), the amount of virtualcontent displayed in the view of the three-dimensional environment(e.g., view 7318 in 7H) is increased compared to an earlier state (e.g.,view 7316 in FIG. 7G), and the visual prominence of the representationof the physical environment is decreased. More specifically, in FIG. 7H,the representations 7004′ and 7006′ of the walls 7004 and 7006 arereplaced or obscured by the display of virtual content 7320 and 7322(e.g., visual effects that visually obscures the portion of therepresentation of the physical environment to which the visual effectsare applied, virtual surfaces, virtual objects, virtual scenery, etc.),and at least a portion of the surface of the representation 7010′ isreplaced or obscured by the display of the virtual content 7324 (e.g.,visual effects that visually obscures the portion of the representationof the physical environment to which the visual effects are applied,virtual surfaces, virtual objects, virtual scenery, etc.) as well. Asshown in FIG. 7I following FIG. 7H, when the values of the biometricdata change away from meeting the preset criteria (e.g., as indicated bythe decreased length of bar 7312 receding from the position of theindicator 7326), the amount of virtual content displayed in the view ofthe three-dimensional environment (e.g., view 7328 in 7I) is decreasedcompared to an earlier state (e.g., view 7318 in FIG. 7H), and thevisual prominence of the representation of the physical environment isincreased again (e.g., optionally, still lower than the state shown inFIG. 7G). More specifically, in FIG. 7I, the representation 7006′ of thewall 7006 is redisplayed after the virtual content 7332 is removed, therepresentation 7004′ of the wall 7004 is partially redisplayed when thevirtual content 7320 is reduced in visual prominence (e.g., visualeffects that visually obscures the portion of the representation of thephysical environment to which the visual effects are applied are reducedin magnitude, virtual surfaces and virtual objects are shrunken,removed, reduced in number, or made more translucent, etc.). The visualprominence of the portion of the surface of the representation 7010′that was replaced or obscured by the display of the virtual content 7324is increased by changes made to the virtual content 7324 (e.g., mademore translucent, less opaque, includes less amount of distortion forthe representation 7010′, etc.) as well. In some embodiments, before thepreset criteria for transitioning to the preset higher level ofimmersion are met (e.g., before the threshold indicated by the indicator7326 is met by the biometric data corresponding to the user, or beforeother criteria are met by the biometric data, etc.), the computer systemcontinuously or periodically adjust the visual balance between virtualcontent and the representation of the physical environment in thecurrently displayed view of the three-dimensional environment (e.g.,increasing visual prominence of the virtual content relative to therepresentation of the physical environment, decreasing visual prominenceof the virtual content relative to the representation of the physicalenvironment, etc.) in accordance with the biometric data, as thebiometric data is updated based on the current state of the user.

In FIG. 7J, the computer system detects that the updated biometric datacorresponding to the user meets the preset criteria for transitioninginto the preset higher level of immersion (e.g., an augmented realityenvironment, an augmented virtuality environment, a virtual realityenvironment, etc.) that has a higher level of immersion as compared tothose displayed before the preset criteria are met by the biometric data(e.g., the views 7316, 7318, 7328 in FIGS. 7G-7I, etc.), and thecomputer system transitions to displaying the three-dimensionalenvironment with the preset higher level of immersion (e.g., displayingthe view 7334 in FIG. 7J, or another view of the three-dimensionalenvironment with the preset higher level of immersion, etc.). In thisexample, as shown in FIG. 7J, the computer system has increased thevisual prominence of virtual content, and further decreased the visualprominence of the representation of the physical environment, such thatonly hints of the physical environment are still visible in thethree-dimensional environment (e.g., the structural relationshipsbetween the walls and floor, the presence of a physical object, etc.)through the visual characteristics of the virtual content (e.g., thevirtual content 7322, 7320, 7330, and 7324 that visually obscures therepresentations 7006′, 7004′, 7008′ and 7010′ of the walls 7006, 7004,the floor 7008, and the physical object 7010 in the view 7334 in FIG.7J). In some embodiments, the computer system further displays virtualobjects in different positions in the three-dimensional environment. Forexample, a virtual object 7332 is displayed at a position thatcorresponds to the location of the physical object 7010 in the physicalenvironment, a virtual object 7326 is displayed at a position thatcorresponds to a location on the floor 7008, and other virtual objectmay be displayed at positions that correspond to free space in thephysical environment or independent of the state of the physicalenvironment, etc. In some embodiments, after the preset criteria fortransitioning into the preset higher level of immersion are met, thecomputer system abruptly increases the amount of virtual content in thecurrently displayed view of the three-dimensional environment ordisplays a completely new environment corresponding to thecomputer-generated experience (e.g., a new virtual world, a new scene,etc.). In some embodiments, after the preset criteria are met and thecomputer system displays the three-dimensional environment with thepreset higher level of immersion, in accordance with a determinationthat the preset criteria are no longer met by the updated biometricdata, the computer system gradually adjust the level of immersion bywhich the three-dimensional environment is displayed based on thechanges in the biometric data, as show in FIGS. 7H and 7I. In someembodiments, after the preset criteria are met and the computer systemdisplays the three-dimensional environment with the preset higher levelof immersion, in accordance with a determination that the presetcriteria are no longer met by the updated biometric data, the computersystem abruptly switches back to displaying the three-dimensionalenvironment with the lower level of immersion (e.g., as shown in FIG.7G).

In some embodiments, the preset criteria are met in accordance with adetermination that the heart rate is lower than a first threshold heartrate, the breathing rate is lower than a first threshold breathing rate,the blood pressure is lower than a first threshold blood pressure,movement of the user is below a first threshold amount of movementduring the threshold amount of time, body temperature of the user islower than a first threshold body temperature, a metric of stress levelis below a first threshold stress level, a metric corresponding touser's mood indicates that the user is relaxed and happy, etc.

In some embodiments, the view of the three-dimensional environment thatis shown with the low level of immersion (e.g., as shown in FIG. 7G, oranother view of the three-dimensional environment, etc.) is displayedwhen the display generation component of the computer system is firstturned on or put on the user's head or in front of the user's eyes, andno virtual elements or a minimal amount of virtual elements aredisplayed in the three-dimensional environment. This allows the user tostart from a view of the three-dimensional environment that is verysimilar to the direct view of the real world without the displaygeneration component blocking the user's eyes. In some embodiments, theview of the three-dimensional environment corresponding to the low levelof immersion is a view of a user interface or environment (e.g., atwo-dimensional environment, a three-dimensional environment, etc.) ofan application or computer-generated experience that is displayed in atwo-dimensional window or confined in a viewport displayed at a positionrelative to the representation of the physical environment. In someembodiments, the view of the three-dimensional environment that is shownwith the low level of immersion (e.g., as shown in FIG. 7G, or anotherview of the three-dimensional environment, etc.) is displayed when theapplication or computer-generated experience is first launched orstarted by the user, and the full spatial extent of the application orexperience are not yet displayed in the three-dimensional environment.This allows the user to start from a view of the three-dimensionalenvironment that is not very immersive and viewed in the context of theview of the real world.

In some embodiments, the virtual content (e.g., virtual wallpaper,virtual objects, virtual surfaces, virtual scenery, virtualthree-dimensional environment, etc.) that is displayed by the computersystem at least partially blocks or obscures the view of the physicalenvironment. In some embodiments, when displaying the view of thethree-dimensional environment with the preset higher level of immersion,the computer system replaces or blocks the view of a first class ofphysical objects or surfaces (e.g., front wall, front wall and ceiling,etc.) with newly displayed virtual element or newly displayed portion ofan existing virtual element. In some embodiments, an animated transitionis displayed to show the virtual elements gradually expanding orbecoming more opaque and saturated to cover or block the view of thefirst class of physical objects or surfaces. In some embodiments, whendisplaying the view of the three-dimensional environment with the presethigher level of immersion, the computer system adds virtual elements tothe three-dimensional environment, without replacing any whole class ofphysical elements. In some embodiments, the virtual elements that areadded include, optionally, a user interface object, such as a menu(e.g., menu of application, documents, etc.), a control (e.g., displaybrightness control, display focus control, etc.), or other objects(e.g., a virtual assistant, a document, media item, etc.) that can bemanipulated by user inputs or provides information or feedback in thethree-dimensional environment. In some embodiments, the virtual elementsthat are added include, optionally, non-interactive objects or surfacesthat cannot be manipulated by user inputs, and serves to provide thelook and feel of the three-dimensional environment that replaces thelook and feel of the physical environment. In some embodiments, thevirtual content that is displayed by the computer system includes avisual effect that at least partially blocks or obscures the view of thephysical environment (e.g., fade out, blurs, dims, etc. therepresentation of the physical environment, etc.).

In some embodiments, in accordance with a determination that thebiometric data is updated and the updated biometric data meets presetcriteria for transitioning to displaying the three-dimensionalenvironment with an even higher level of immersion, the computer systemincreases the visual prominence of the virtual content corresponding tothe computer-generated experience and reduces visual stimuli from thephysical environment to another level corresponding to the even higherlevel of immersion. For example, in some embodiments, the computersystem causes an additional class of physical objects or surfaces to bereplaced, obscured, and/or blocked by the newly displayed virtualelement or newly displayed portion of an existing virtual element. Insome embodiments, an animated transition is displayed to show thevirtual elements gradually expanding or becoming more opaque andsaturated to cover or block the view of the additional class of physicalobjects and surfaces.

In some embodiments, the three-dimensional environment is an environmentof a computer-generated mediation experience, and as the biometric dataindicates that the user has achieved the level of concentration,relaxation, focus, etc. required to enter a deeper state of meditativeexperience, the computer system transforms the currently displayed viewof the environment into a more immersive environment, e.g., withexpanded spatial range (e.g., width, depth, angle, etc.) and visualprominence of the virtual content corresponding to the meditativeexperience and reduced spatial range and visual prominence of therepresentation of the physical environment.

In some embodiments, with the increased level of immersion with whichvisual content of the computer-generated experience is displayed, thecomputer system also increases the level of suppression of sounds of thephysical environment perceivable by the user through actions of theaudio output devices of the computer system and/or increases the levelof immersion of the audio content of the computer-generated experience(e.g., increasing volume, changing to a spatial audio output mode from astereo audio output mode or surround sound output mode, or from a stereoaudio output mode to a surround sound output mode, etc.) that is outputby the audio output devices.

In some embodiments, the computing system is configured to displayvisual component of CGR content via a display generation component withtwo or more levels of immersion. In some embodiments, the computersystem displays the visual component of the CGR content with at least afirst level of immersion, a second level of immersion, and a third levelof immersion. In some embodiments, the computer system displays thevisual component of the CGR content with at least two levels ofimmersion, respectively providing a less immersive visual experience anda more immersive visual experience relative to each other. In someembodiments, the computing system causes the visual content displayedvia the display generation component to transition between the differentlevels of immersion in response to the biometric data corresponding tothe user meeting different sets of criteria. In some embodiments, thefirst, second, and third levels of immersion correspond to increasingamount of virtual content corresponding to the CGR experience that ispresent in the CGR environment and/or decreasing amount ofrepresentations of the surrounding physical environment present in theCGR environment. In some embodiments, first, second, and third levels ofimmersion correspond to different modes of content display that haveincreasing image fidelity (e.g., increasing pixel resolution, increasingcolor resolution, increasing color saturation, increasing luminance,increasing opacity, increasing image details, etc.) and/or spatialextent (e.g., angular extent, spatial depth, etc.) for the visualcomponent of the computer-generated content, and/or decreasing imagefidelity and/or spatial extent for the representation of the surroundingphysical environment. In some embodiments, the first level of immersionis a pass-through mode where the physical environment is fully visibleto the user through the display generation component (e.g., as a cameraview of the physical environment or through a transparent orsemi-transparent portion of the display generation component). In someembodiments, the visual CGR content presented in the pass-through modeincludes the pass-through view of the physical environment with aminimal amount of virtual elements concurrently visible as the view ofthe physical environment or with only virtual elements that areperipheral (e.g., indicators and controls displayed in the peripheralregion of the display) to the user's view of the physical environment.For example, a view of the physical environment occupies the central andmajority region of the field of view provided by the display generationcomponent, and only a few controls (e.g., the title of the movie, theprogress bar, playback control (e.g., play button), etc.) are displayedin the peripheral region of the field of view provided by the displaygeneration component. In some embodiments, the first level of immersionis a pass-through mode where the physical environment is fully visibleto the first user through the display generation component (e.g., as acamera view of the physical environment or through a transparent portionof the display generation component), and the visual CGR content isdisplayed in a virtual window or frame that overlays, replacing displayof, or blocking the view of, etc. a portion of the representation of thephysical environment. In some embodiments, the second level of immersionis a mixed reality mode where the pass-through view of the physicalenvironment is augmented with virtual elements generated by the computersystem, where the virtual elements occupy the central and/or majorityregion of the user's field of view (e.g., the virtual content isintegrated with the physical environment in the view of thecomputer-generated environment). In some embodiments, the second levelof immersion is a mixed reality mode where the pass-through view of thephysical environment is augmented with a virtual window, viewport, orframe that overlays, replacing display of, or blocking the view of, etc.a portion of the representation of the physical environment, and thathas additional depth or spatial extent that are revealed when thedisplay generation component is moved relative to the physicalenvironment. In some embodiments, the third level of immersion is anaugmented reality mode where virtual content is displayed in athree-dimensional environment with a representation of the physicalenvironment, and virtual objects are distributed throughout thethree-dimensional environment at positions corresponding to differentlocations of the physical environment. In some embodiments, the thirdlevel of immersion is a virtual reality mode where virtual content isdisplayed in a three-dimensional environment without a representation ofthe physical environment. In some embodiments, the different levels ofimmersion described above represents increasing levels of immersionrelative to one another.

In some embodiments, the computer system selects the audio output modefor outputting the audio content of a computer-generated experience(e.g., an application, a communication session, a movie, a video, agame, etc.) in accordance with the level of immersion with which thevisual content of the computer-generated experience is being displayedby the display generation component. In some embodiments, when the levelof immersion with which the visual content is displayed increases (e.g.,from the first level of immersion to the second level of immersion, fromthe first level of immersion to the third level of immersion, or fromthe second level of immersion to the third level of immersion, etc.),the computer system switches the audio output mode from a less immersiveoutput mode to a more immersive output mode (e.g., from a first audiooutput mode to a second audio output mode, or from a first audio outputmode to a third audio output mode, or from a second audio output mode toa third audio output mode, etc., where the first audio output mode, thesecond audio output mode, and the third audio output mode correspond toaudio output with increasing levels of immersion). As described herein,a spatial audio output mode corresponds to a higher level of immersionthan a stereo audio output mode and a mono audio output mode. A spatialaudio output mode corresponds to a higher level of immersion than asurround sound output mode. A surround sound output mode corresponds toa higher level of immersion than a stereo audio output mode and a monoaudio output mode. A stereo audio output mode corresponds to a higherlevel of immersion than a mono audio output mode. In some embodiments,the computer system selects an audio output mode from multiple availableaudio output modes, e.g., a mono audio output mode, a stereo audiooutput mode, a surround sound output mode, a spatial audio output mode,etc. based on the level of immersion with which visual content of acomputer-generated experience is being provided via the displaygeneration component.

FIGS. 7K-7M are block diagrams illustrating aggregating the effects ofmultiple types of the sensory adjustment provided by a computer systemwhen displaying a view of an environment that includes a representationof a physical environment, in accordance with some embodiments.

In some embodiments, the computer system provides multiple types ofsensory adjustment functions that enhance the user's ability to perceivedifferent aspects of a physical environment that may not be easilyperceivable without the aid of special equipment or the computer system.Instead of allowing the user to only use a single type of sensoryadjustment function when viewing a portion of a physical environment ata time, the computer system aggregates the effects of two or more typesof sensory enhancement functions on a representation of the portion ofthe physical environment, such that features and characteristics presentin the portion of the physical environment that were previously hiddenin the view of the physical environment provided by the computer systemmay be revealed.

In some embodiments, when the computer system displays athree-dimensional environment that includes a representation of aphysical environment via a display generation component (e.g., displaygeneration component 7100, or another type of display generationcomponent such as an HMD, etc.), the computer system optionally usessensor input or information that corresponds to the currently displayedportion of the physical environment to augment and adjust therepresentation of the physical environment, such that the user canperceive the portion of the physical environment with sensoryinformation that is not available to the user when the user views theportion of the physical environment without the aid of the displaygeneration component.

In FIG. 7K, the computer system displays a view 7340 of athree-dimensional environment that includes a first representation of afirst portion of the physical environment. In the view 7340, the firstrepresentation of the first portion of the physical environmentcorresponds to an appearance of the first portion of the physicalenvironment without sensory adjustments made by the computer system. Insome embodiments, the first representation of the first portion of thephysical environment corresponds to a view of the first portion of thephysical environment that is captured by a color camera that has a firstlevel of imaging sensitivity that corresponds to an average color andintensity detection within the range of normal human sensory perception.In some embodiments, the first representation of the first portion ofthe physical environment corresponds to a view of the first portion ofthe physical environment through a transparent portion of the displaygeneration component and is not enhanced and adjusted by the computersystem.

In some embodiments, the computer system provides a plurality ofaffordances (e.g., hardware controls 7354, 7356, and 7358, userinterface elements that are displayed in the three-dimensionalenvironment, etc.) for activating respective ones of a plurality ofsensory adjustment functions provided by the computer system. In someembodiments, the computer system activates the respective ones of theplurality of sensory adjustment functions in a sequence or incombination, in accordance with a user's activation inputs (e.g., buttonpress inputs, tap inputs, gesture inputs, touch inputs, gaze inputs,selection input, a combination thereof, etc.) directed to theaffordances corresponding to the respective ones of the plurality ofsensory adjustment functions. In some embodiments, a respective one ofthe plurality of sensory adjustment functions is optionally activated bya preset input (e.g., a gesture input, a touch input, a voice command,etc.) without requiring presence of a corresponding hardware affordanceassociated with the computer system or a corresponding user interfacecontrol in the three-dimensional environment.

In this example, as shown in FIG. 7K, the first representation of thefirst portion of the physical environment includes a view from inside ofa room toward a window on a wall of the room. This example isnon-limiting, and the first portion of the physical environment may beany indoor or outdoor environment, in accordance with variousembodiments. In this example, the first representation of the firstportion of the physical environment includes a representation 7344′ ofthe wall, the representation 7346′ of the window, a representation 7348′of a hill outside of the window at a first distance from the window, anda representation 7350′ of a tree near the top of the hill at a seconddistance away from the window. The representation 7348′ of the hill andthe representation 7350′ of the tree occupy a small portion of the fieldof view provided by the display generation component 7100 because thehill and the tree are at large distances away from the displaygeneration component, and the representation 7348′ of the hill and therepresentation 7350′ of the tree are far away from the viewpointcorresponding to the currently displayed view of the three-dimensionalenvironment (e.g., the respective distances between the viewpoint andthe representations 7348′ and 7350′ correspond to the respectivedistances from the user's eyes (or the display generation component) tothe hill and the tree).

In FIG. 7L, the computer system detects a user input that activates afirst sensory adjustment function of a plurality of sensory adjustmentfunctions provided by the computer system. For example, the computersystem detects that the hardware affordance 7354 is activated by auser's input, that a user interface object corresponding to the firstsensory adjustment function is activated or selected by a user's input,that a gesture input, voice command, and/or a touch input, etc. meetingthe criteria for activating the first sensory adjustment function isprovided by the user, etc. In response, the computer system displays asecond view 7361 of the three-dimensional environment that includes asecond representation of a second portion of the physical environment,where the second portion of the physical environment is included withinthe first portion of the physical environment (e.g., is all or asub-portion of the first portion of the physical environment shown inFIG. 7K, or a portion of the physical environment that was shown beforethe detection of the input that activated the first sensory adjustmentfunction, etc.). In the second view 7361 of the three-dimensionalenvironment, as shown in the example in FIG. 7L, the display property ofthe representation 7350″ of the tree is adjusted relative to therepresentation 7350′ of the tree shown in the first view 7340 of thethree-dimensional environment in accordance with the operation of thefirst sensory adjustment function. For example, if the first sensoryadjustment function is simulated telescope vision that reduces the focusdistance of objects such that they appear closer to the user, as shownin FIG. 7L, the representation 7350″ of the tree appears to be locatedmuch closer to the viewpoint than the second distance as shown in FIG.7K (e.g., the adjusted distance is one fifth of the second distance, theadjusted distance is one tenth of the second distance, the adjusteddistance is a distance that is selected based on the second distanceand/or a preset fraction of the maximum power of the simulated telescopevision, etc.). Similarly, the representation 7348″ of the hill alsoappears to be located much closer to the user than the first distance asshown in FIG. 7K (e.g., the adjusted distance is one fifth of the firstdistance, the adjusted distance is one tenth of the first distance, theadjusted distance is a distance that is selected based on the firstdistance and/or a preset fraction of the maximum power of the simulatedtelescope function, etc.). In this example, the viewpoint or the virtualposition of the user in the view 7361 is moved to the position of thewindow in the view 7340, in accordance with some embodiments. In thisexample, the viewpoint or the virtual position of the user in the view7361 is still based on the actual location of the user and/or thedisplay generation component in the physical environment, in accordancewith some embodiments.

In some embodiments, the computer, when applying the first sensoryadjustment function, selects a target portion of the physicalenvironment based on a location of the user's gaze directed to thecurrently view of the three-dimensional environment. For example, asshown in FIG. 7K, the computer system detects that the user's gaze 7352is directed to the representation 7350′ of the tree in the first view7340 of the three-dimensional environment, and selects a portion of thephysical environment that includes the tree from the first portion ofthe physical environment as the second portion of the physicalenvironment to which the first sensory adjustment function is applied.

In some embodiments, the simulated telescope vision is an illustrativeexample of a first type of sensory adjustment function provided by thecomputer system, and may be replaced by another type of sensoryadjustment function that is provided by the computer system and selectedby the user's input.

In FIG. 7M, while the computer system is displaying the second view 7361of the three-dimensional environment that includes the secondrepresentation of the physical environment that has been adjusted inaccordance with the operation of the first sensory adjustment functionactivated by the user's input, the computer system detects a second userinput that activates a second sensory adjustment function of theplurality of sensory adjustment functions that is different from thefirst sensory adjustment function. For example, the computer systemdetects that the hardware affordance 7356 is activated by a user'sinput, that a user interface object corresponding to the second sensoryadjustment function is activated or selected by a user's input, that agesture input, voice command, and/or a touch input, etc. meeting thecriteria for activating the second sensory adjustment function isprovided by the user, etc. In response, the computer system displays athird view 7364 of the three-dimensional environment that includes athird representation of a third portion of the physical environment,where the third portion of the physical environment is included withinthe second portion of the physical environment (e.g., is all or asub-portion of the second portion of the physical environment shown inFIG. 7L, or a portion of the physical environment that was shown beforethe detection of the input that activated the second sensory adjustmentfunction, etc.). In the third view 7364 of the three-dimensionalenvironment, as shown in the example in FIG. 7M, the display property ofthe representation 7350′″ of the tree is further adjusted relative tothe representation 7350″ of the tree shown in the second view 7361 ofthe three-dimensional environment in accordance with the operation ofthe second sensory adjustment function. For example, if the secondsensory adjustment function is simulated heat vision that presents colorand/or intensity variations in accordance with the temperature and/orthermal radiation variations, as shown in FIG. 7M, the representation7350′″ of the tree appears to be have a different color and/or intensityrelative to the background environment in the third view 7364, and thedisplay property of portions 7366′ and 7368′ of the representation7350′″ are further adjusted based on the temperature of those portionsof the tree relative to other portions of the tree in the physicalenvironment (e.g., as detected by the thermal imaging sensors or othersensors that are in communication with the computer system, as indicatedby thermal data transmitted to or retrieved by the computer system fromanother computer system, etc.). For example, the higher temperature ofthose portions represented by portions 7366′ and 7368′ likely revealsmall animals or objects that radiate more heat or have highertemperatures than the tree itself. The portions 7366′ and 7368′ in therepresentation 7350′″ have display properties that are generated basedon the operations of both the first sensory adjustment function and thesecond sensory adjustment function on the original first representation7350′ of the tree as shown in FIG. 7K.

In some embodiments, the computer system, when applying the secondsensory adjustment function, selects a target portion of the physicalenvironment based on a location of the user's gaze directed to thecurrently displayed view of the three-dimensional environment. Forexample, as shown in FIG. 7L, the computer system detects that theuser's gaze 7360 is directed to the representation 7350″ of the tree inthe second view 7361 of the three-dimensional environment, and selects aportion of the physical environment that includes the tree from thesecond portion of the physical environment as the third portion of thephysical environment to which the first sensory adjustment function andthe second sensory adjustment function are both applied.

In some embodiments, the simulated heat vision is an illustrativeexample of a second type of sensory adjustment function provided by thecomputer system, and may be replaced by another type of sensoryadjustment function that is provided by the computer system and selectedby the user's input.

In some embodiments, a first display property (e.g., resolution, zoomlevel, magnification, color distribution, intensity distribution, focusdistance, etc.) is adjusted relative to a baseline representation of arespective portion of the physical environment (e.g., the portions ofthe representation 7350′ of the tree in FIG. 7K that correspond to theportions 7366′ and 7368′ in FIG. 7M, another portion of the physicalenvironment, etc.) in accordance with a first type of computer-generatedsensory adjustment (e.g., binocular vision, telescope vision, microscopevision, night vision, heat vision, etc.) to obtain a first adjustedrepresentation of the respective portion of the physical environment(e.g., the portions of the representation 7350″ tree in FIG. 7L thatcorrespond to the portions 7366′ and 7368′ in FIG. 7M, or adjustedrepresentation of another portion of the physical environment, etc.),and a second display property (e.g., resolution, zoom level,magnification, color distribution, intensity distribution, focusdistance, etc.) is adjusted relative to the second representation of thephysical environment to obtain a third representation of the respectiveportion of the physical environment (e.g., the portions 7366′ and 7368′of the representation 7350′ of the tree in FIG. 7M, or further adjustedrepresentation of another portion of the physical environment, etc.) inaccordance with the second type of computer-generated sensoryadjustment. In some embodiments, the second display property has thesame values in the first representation and the second representation insome combinations of the first type and second type of sensoryadjustment functions; and the second display property has differentvalues in the first representation and the second representation in somecombinations of the first type and second type of sensory adjustmentfunctions.

In some embodiments, the computer system allows the representation ofthe physical environment to be adjusted further based on a third sensoryadjustment function (e.g., the sensory adjustment function that can beactivated by interaction with the affordance 7358, a user interfaceobject, a gesture input, a voice command, etc. corresponding to thethird sensory adjustment function, etc.). In some embodiments, whiledisplaying the third view 7364 of the three-dimensional environment thatincludes the third representation of the physical environment, thecomputer system detects a third user input that corresponds to a requestto activate the third type of computer-generated sensory adjustment(e.g., binocular vision, microscope vision, night vision, heat vision,color filter, etc.) that is different from the first type and secondtype of sensory adjustment functions. In response, the computer systemdisplays a fourth view of the three-dimensional environment thatincludes a fourth representation of a fourth portion of the physicalenvironment (e.g., all or a portion of the third portion of the physicalenvironment), wherein the fourth representation of the physicalenvironment has the first display property (e.g., resolution, zoomlevel, magnification, color distribution, intensity distribution, focusdistance, etc.) adjusted relative to the first representation of thefourth portion of the physical environment in accordance with the firsttype of sensory adjustment function, the second display property (e.g.,resolution, zoom level, magnification, color distribution, intensitydistribution, focus distance, etc.) adjusted relative to the secondrepresentation of the fourth portion of the physical environment inaccordance with the second type of sensory adjustment function, and athird display property (e.g., resolution, zoom level, magnification,color distribution, intensity distribution, focus distance, etc.) thatis adjusted relative to the third representation of the physicalenvironment of the fourth portion of the physical environment inaccordance with the third type of sensory adjustment function.

In some embodiments, the first sensory adjustment function includessimulated telescope vision (e.g., binocular vision, monocular vision,telescope vision, etc.) (e.g., reducing focus distance of objects suchthat they appear closer to the user) for viewing distant physicalobjects, and the second sensory adjustment function includes simulatedmicroscope vision for magnifying nearby physical objects.

In some embodiments, the first sensory adjustment function includessimulated telescope vision (e.g., reducing focus distance of objectssuch that they appear closer to the user) for viewing distant physicalobjects, and the second sensory adjustment function includes simulatednight vision (e.g., high sensitivity in low light conditions, brightnessof objects are visually enhanced, small variations in brightness aremagnified, etc.) for viewing physical objects under low lightconditions.

In some embodiments, the first sensory adjustment function includessimulated telescope vision (e.g., reducing focus distance of objectssuch that they appear closer to the user) for viewing distant physicalobjects, and the second sensory adjustment function includes modifying aview of physical objects with a filter (e.g., color filter, lightfrequency filter, intensity filter, a motion filter, etc.).

In some embodiments, the first sensory adjustment function includessimulated telescope vision (e.g., reducing focus distance of objectssuch that they appear closer to the user) for viewing distant physicalobjects, and the second sensory adjustment function includes selectiveaudio enhancement (e.g., enhancing volume, selectivelyenhancing/suppressing certain sound frequencies, etc.) for soundscorresponding to a subset of physical objects (e.g., a selected subsetof all sound producing physical objects, physical objects that are inthe center of the current field of view, etc.) in the physicalenvironment.

In some embodiments, concurrently with displaying the thirdrepresentation of the physical environment, the computer system outputssounds that correspond to a portion of the physical environment visiblein the third representation of the physical environment, wherein thesounds are selectively enhanced (e.g., increased in volume, withmodifications to the amplitudes of some selected frequencies, etc.)relative to sounds from sources outside of the portion of the physicalenvironment.

In some embodiments, concurrently with displaying the thirdrepresentation of the physical environment, the computer system displaystextual output corresponding to speech coming from a portion of thephysical environment visible in both the second representation and thirdrepresentation of the physical environment, wherein the speech isselectively enhanced relative to sounds from sources outside of theportion of the physical environment.

In some embodiments, the first sensory adjustment function includessimulated microscope vision for magnifying nearby physical objects, andthe second sensory adjustment function includes simulated heat vision(e.g., high sensitivity to temperature variations, presenting colorand/or intensity variations in accordance with temperature and/orthermal radiation variations, etc.) for viewing physical objects withdifferent thermal radiation profiles.

In some embodiments, the first sensory adjustment function includessimulated night vision (e.g., high sensitivity in low light conditions,brightness of objects are visually enhanced, small variations inbrightness are magnified, etc.) for viewing physical objects under lowlight conditions, and the second sensory adjustment function includessimulated telescope vision (e.g., reducing focus distance of objectssuch that they appear closer to the user) for viewing distant physicalobjects.

In some embodiments, the first sensory adjustment function includessimulated night vision (e.g., high sensitivity in low light conditions,brightness of objects are visually enhanced, small variations inbrightness are magnified, etc.) for viewing physical objects under lowlight conditions, and the second sensory adjustment function includessimulated microscope vision for magnifying nearby physical objects.

In some embodiments, the first sensory adjustment function includessimulated night vision (e.g., high sensitivity in low light conditions,brightness of objects are visually enhanced, small variations inbrightness are magnified, etc.) for viewing physical objects under lowlight conditions, and the second sensory adjustment function includessimulated heat vision (e.g., high sensitivity to temperature variations,presenting color and/or intensity variations in accordance withtemperature and/or thermal radiation variations, etc.) for viewingphysical objects with different thermal radiation profiles.

In some embodiments, the first sensory adjustment function includessimulated night vision (e.g., high sensitivity in low light conditions,brightness of objects are visually enhanced, small variations inbrightness are magnified, etc.) for viewing physical objects under lowlight conditions, and the second sensory adjustment function includesand the second type of computer-generated sensory adjustment includesselective audio enhancement (e.g., enhancing volume, selectivelyenhancing/suppressing certain sound frequencies, etc.) for soundscorresponding to a subset of physical objects (e.g., a selected subsetof all sound producing physical objects, physical objects that are inthe center of the current field of view, etc.) in the physicalenvironment.

In some embodiments, the first sensory adjustment function includessimulated heat vision (e.g., high sensitivity to temperature variations,presenting color and/or intensity variations in accordance withtemperature and/or thermal radiation variations, etc.) for viewingphysical objects with different thermal radiation profiles and thesecond sensory adjustment function includes simulated telescope vision(e.g., reducing focus distance of objects such that they appear closerto the user) for viewing distant physical objects.

In some embodiments, the first sensory adjustment function includessimulated heat vision (e.g., high sensitivity to temperature variations,presenting color and/or intensity variations in accordance withtemperature and/or thermal radiation variations, etc.) for viewingphysical objects with different thermal radiation profiles and thesecond sensory adjustment function includes simulated microscope visionfor magnifying nearby physical objects.

In some embodiments, the first sensory adjustment operation includessimulated heat vision (e.g., high sensitivity to temperature variations,presenting color and/or intensity variations in accordance withtemperature and/or thermal radiation variations, etc.) for viewingphysical objects with different thermal radiation profiles, and thesecond sensory adjustment operation includes simulated night vision(e.g., high sensitivity in low light conditions, brightness of objectsare visually enhanced, small variations in brightness are magnified,etc.) for viewing physical objects under low light conditions.

In some embodiments, the first sensory adjustment function includessimulated heat vision (e.g., high sensitivity to temperature variations,presenting color and/or intensity variations in accordance withtemperature and/or thermal radiation variations, etc.) for viewingphysical objects with different thermal radiation profiles, and thesecond sensory adjustment operation includes selective audio enhancement(e.g., enhancing volume, selectively enhancing/suppressing certain soundfrequencies, etc.) for sounds corresponding to a subset of physicalobjects (e.g., a selected subset of all sound producing physicalobjects, physical objects that are in the center of the current field ofview, etc.) in a physical environment.

In some embodiments, the order by which a plurality of selected sensoryadjustment functions selected by a user are applied to the baselinerepresentation of a portion of the physical environment is adjusted bythe computer system based on one or more preset constrains and are,optionally, different from the order by which these sensory adjustmentfunctions are activated by the user. For example, in some embodiments,adjustments corresponding to simulated telescope vision is performedprior to adjustments corresponding to other types of sensoryadjustments, because it would reduce the portion of the physicalenvironment that the other types of sensory adjustment need to beperformed for the purposes of presenting the final result to the user.In some embodiments, the computer system observes the order that thedifferent types of sensory adjustment functions are activated by theuser, and presents the intermediate result obtained in response to eachadditional sensory adjustment that is activated by the user.

FIGS. 7N-7P are block diagrams illustrating selectively displayingvirtual content that corresponds to a respective type of exercise in aview of a three-dimensional environment in accordance with adetermination that the portion of the physical environment in the viewof the three-dimensional environment corresponds to the respective typeof exercise, in accordance with some embodiments.

In some embodiments, the computer system displays virtual content (e.g.,virtual open water 7406, virtual hiking trail 7412, etc.) (e.g., virtualscenery, visual and functional enhancements of the exercise equipment,user interfaces, health and score boards, etc.) that corresponds to arespective type of exercise (e.g., rowing, hiking, etc.) in accordancewith a determination that the physical location (e.g., location of thephysical object 7404, location of the physical object 7402, etc.)represented in a view of a three-dimensional environment (e.g., view7408, view 7410, etc.) is associated with the respective type ofexercise (e.g., rowing, hiking, etc.). For example, as the user and thedisplay generation component (e.g., user 7002 and display generationcomponent 7100, or another user with another type of display generationcomponent such as an HMD, etc.) move from location to location in thereal world (e.g., in the scene 105, or in another physical environment,etc.), the virtual content shown in the view of the three-dimensionalenvironment is adjusted to correspond to the type of exercise that isassociated with the current location of the user and the displaygeneration component. In some embodiments, when a location is associatedwith multiple types of exercise, the computer system selects a type ofexercise from the multiple types of exercises that are associated withthe location based on other contextual information (e.g., movement ofthe user, engagement of the user with the objects at the location,etc.), and displays the visual content corresponding to the selectedtype of exercise.

FIG. 7N shows, in part (A), that a user 7002 is located in a physicalenvironment (e.g., scene 105, or another physical environment, etc.).The user 7002 may be located in a different physical environment that isan outdoor environment or an indoor environment, or moves between anindoor and an outdoor environment, etc. The user 7002 views the physicalenvironment through a field of view provided via the first displaygeneration component (e.g., display generation component 7100, anothertype of display generation component such as an HMD, etc.). The physicalenvironment includes physical surfaces (e.g., walls 7004 and 7006, floor7008, other physical surfaces, etc.) and one or more physical objects(e.g., exercise equipment 7402, 7404, other physical objects, etc.). Insome embodiments, the physical environment is a building that includesmultiple rooms or sections that are separate from one another thatcannot be viewed by the user at the same time. In some embodiments, thephysical environment includes multiple areas that are separate from eachother, such as rooms in separate buildings, different parks, differentgeographical regions, etc. In some embodiments, the physical environmentis an outdoor environment that include outdoor physical objects andsurfaces, such as roads, trees, sky, open water, rocks, mountains,vehicles, animals, people, etc. In some embodiments, the computer systemstores information and/or implements rules and artificial intelligenceto determined one or more types of exercises (e.g., indoor exercise,indoor sports, outdoor exercises, outdoor sports, physical activitiesthat promote health and physical capabilities, physical rehabilitationand therapy, etc.) that are associated with a respective location thatis in the user's physical environment (e.g., within the user's field ofview through the display generation component, in a threshold vicinityof the user (e.g., within 5 meters, within a few steps, etc.), etc.). Insome embodiments, the computer system determines the type of exercisethat is associated with a respective location based on the types ofphysical objects that are present at the respective location. In someembodiments, the computer system determines the type of exercise that isassociated with a respective location based on the types of environmentor setting that are present at the respective location. In someembodiments, the computer system determines the type of exercise that isassociated with a respective location based on other types of markersand signals, or combinations of information that are present at therespective location.

In FIG. 7N, part (B), the computer system displays a first view 7405 ofa three-dimensional environment that includes a representation of thephysical environment of the user 7002. In some embodiments, the firstview 7405 of the three-dimensional environment is a reality view with novirtual elements or minimal virtual elements, as shown in FIG. 7N(B). Inthis example, the first view 7405 includes representations of physicalsurfaces (e.g., representations 7004′ and 7006′ of the walls 7004 and7006, representation 7008 of the floor 7008, etc.) and representationsof physical objects (e.g., representation 7402′ of the physical object7402, representation 7404′ of the physical object 7404, etc.), withoutvirtual content. In some embodiments, the first view of thethree-dimensional environment is a reality view (e.g., the view 7405shown in FIG. 7N(B)) with user interface objects for controlling basicfunctions of the computer system (e.g., application icons for launchingdifferent computer-generated experiences, display settings, audiocontrols, etc.). In some embodiments, the first view of thethree-dimensional environment is an augmented reality view displayedwith a low-level of immersion (e.g., displaying user interface objects(e.g., an application launch pad, a welcome user interface, a settingsuser interface) that are not part of a specific application experience(e.g., a health application, a meditation application, a workoutapplication, a game application, etc.), and that on aggregate onlyoccupy a small percentage (e.g., less than 10%, less than 20%, etc.) ofthe user's field of view or are displayed in confined floating windows,etc. In some embodiments, the representation of the physical environmentincluded in the first view 7405 of the three-dimensional environment isa camera view of a portion of the physical environment. In someembodiments, the portion of the physical environment that is shown inthe first view 7405 of the three-dimensional environment changes as theuser moves around the physical environment (e.g., when the user wear'sthe display generation component on his/her head, or holds the displaygeneration component in his/her hand, etc.). In some embodiments, theportion of the physical environment that is shown in the first view 7405of the three-dimensional environment changes as the display generationcomponent is moved around the physical environment. In some embodiments,the representation of the physical environment included in the firstview 7405 of the three-dimensional environment is a view of the physicalenvironment through a transparent portion of the display generationcomponent. In this example, the physical object 7402 is located at afirst location within a first portion of the physical environment shownin the first view 7405 of the three-dimensional environment, and thephysical object 7404 is located at a second location within the firstportion of the physical environment shown in the first view 7405 of thethree-dimensional environment. In some embodiments, the first locationand the second location are not necessarily within the same view of thethree-dimensional environment, and may be located in two separatelocations within the same physical environment or in different physicalenvironments that are completely separate from each other. In thisexample, the physical object 7402 corresponds to equipment or settingcorresponding to a first type of exercise (e.g., running, walking,etc.), and the physical object 7402 corresponds to equipment or settingcorresponding to a second type of exercise (e.g., rowing, boating, waterskiing, etc.).

In FIGS. 7O and 7P, the computer system detects movement of the user7002 in the physical environment, while displaying the first view 7450of the three-dimensional environment. In some embodiments, the portionof the physical environment that is visible within the first view of thethree-dimensional environment changes as the user moves about in thephysical environment. FIG. 7O illustrates a first scenario in which theuser 7002 has moved to the first location that includes the physicalobject 7404 or setting that corresponds to the first type of exercise.FIG. 7O illustrates a second scenario in which the user 7002 has movedto the second location that includes the physical object 7402 or settingthat corresponds to the second type of exercise.

In some embodiments, the movement of the user includes movement of theuser as a whole to a respective location (e.g., the first location thatincludes the first physical object 7404, the second location thatincludes the second physical object 7402, etc.) (e.g., while the user isholding or wearing the display generation component, while a spatialrelationship between the display generation component and the userremains such that the user can continue to view the physical environmentthrough the display generation component, etc.). In some embodiments,the movement of the user includes movement of the user that orients thedisplay generation component or the camera associated with the displaygeneration component to capture a view of the respective location (e.g.,the first location that includes the first physical object 7404, thesecond location that includes the second physical object 7402, etc.)(e.g., while the user is holding or wearing the display generationcomponent, while a spatial relationship between the display generationcomponent and the user remains such that the user can continue to viewthe physical environment through the display generation component,etc.). In some embodiments, the movement of the user further includesmovement that corresponds to manipulation of the physical object(s) atthe respective location (e.g., turning on a piece of exercise equipmentat the respective location, picking up a piece of exercise equipment atthe respective location, start to use the exercise equipment at therespective location, etc.).

As illustrated in FIG. 7O, the user has moved to the first location inthe physical environment that includes the physical object 7404 thatcorresponds to the first type of exercise. In this example, the user hasalso moved into a position relative to the physical object 7404 thatenables the user to start using the physical object 7404 for the firsttype of exercise (e.g., sitting down on the equipment, standing on theequipment, holding one or more portions of the equipment, etc.). In someembodiments, the computer system, optionally, detects that the user hasstarted one or more repetitions of movement that corresponds to thefirst type of exercise (e.g., rowing the ores, pulling on a gear shift,assuming a starting posture, etc.). In response to detecting themovement of the first user to the first location that includes thephysical object 7404 that corresponds to the first type of exercise, andin accordance with a determination that the first location correspondsto the first type of exercise, and, optionally, that the movement of theuser meets a first set of criteria (e.g., criteria corresponding to thefirst location, criteria corresponding to the first type of exercise,etc.), the computer system displays a second view 7408 of thethree-dimensional environment, where the second view 7408 includes firstvirtual content corresponding to the first type of exercise, and a viewof the first virtual content replaces at least a portion of the view ofthe physical environment that includes the first location (e.g., thelocation that includes the physical object 7404 and does not include thephysical object 7402, the location that does not correspond to thesecond type of exercise, etc.). In some embodiments, the first virtualcontent completely replaces the view of the physical environment in thesecond view 7408 of the three-dimensional environment. In someembodiments, the virtual content is displayed overlaying, blocking, orreplacing display of the representation of the physical environment inthe second view 7408 of the three-dimensional environment.

In some embodiments, the computer system determines that the firstlocation corresponds to the first type of exercise in accordance with adetermination that the first location has a first type of exerciseequipment (e.g., rowing machines, boat, etc.) corresponding to the firsttype of exercise. In some embodiments, the computer system determinesthat the first location corresponds to the first type of exercise inaccordance with a determination that the first location is a locationdesigned for (e.g., having appropriate floor surface, structures, etc.for) the first type of exercise (e.g., rowing, meditation, etc.).

As shown in FIG. 7O, part (B), the computer system displays a secondview 7408 of the three-dimensional environment, when the user 7002 hasmoved to the first location that corresponds to the first type ofexercise. In some embodiments, the second view 7408 is an augmentedreality view with more virtual elements corresponding to a firstcomputer-generated experience corresponding to the first location andthe first type of exercise. In some embodiments, the second view 7408 isan augmented reality view showing a preview or start of a firstcomputer-generated experience corresponding to the first location andthe first type of exercise. In some embodiments, the second view 7408 isan augmented reality view displayed with a higher-level of immersion(e.g., displaying user interface objects that are part of a firstspecific application experience corresponding to the first type ofexercise (e.g., virtual hiking trails, virtual scenery, score boards,exercise statistics, controls of changing exercise parameters, etc.),that on aggregate occupy a substantial percentage (e.g., greater than60%, greater than 90%, etc.) of the user's field of view or aredisplayed in a three-dimensional virtual or augmented realityenvironment. In this example, the virtual content displayed in thesecond view of the three-dimensional environment includes virtual openwater 7406 that replaced the view of the representations 7004′, 7006′,and/or 7008′ of various portions of the physical environment that arepotentially within the field of view provided by the display generationcomponent at the first location. In some embodiment, all portions of thephysical environment in the potential field of view provided by thedisplay generation component are replaced or blocked by the display ofthe virtual content. In some embodiments, a portion of the physicalenvironment such as a portion of the user's body, at least a portion ofthe exercise equipment, etc. remain visible in the second view 7408 ofthe three-dimensional environment.

As illustrated in FIG. 7P, the user has moved to the second location inthe physical environment that includes the physical object 7402 thatcorresponds to the second type of exercise. In this example, the userhas also moved into a position relative to the physical object 7402 thatenables the user to start using the physical object 7402 for the secondtype of exercise (e.g., sitting down on the equipment, standing on theequipment, holding one or more portions of the equipment, etc.). In someembodiments, the computer system, optionally, detects that the user hasstarted one or more repetitions of movement that corresponds to thesecond type of exercise (e.g., stepping on the stairs, start pedaling,start walking, etc.). In response to detecting the movement of the userto the second location that includes the physical object 7402 thatcorresponds to the second type of exercise, and in accordance with adetermination that the second location corresponds to the second type ofexercise, and, optionally, that the movement of the user meets a secondset of criteria (e.g., criteria corresponding to the second location,criteria corresponding to the second type of exercise, etc.), thecomputer system displays a third view 7412 of the three-dimensionalenvironment, where the third view 7412 includes second virtual contentcorresponding to the second type of exercise, and a view of the secondvirtual content replaces at least a portion of the view of the physicalenvironment that includes the second location (e.g., the location thatcorresponds to the second type of exercise but not the first type ofexercise, the location that does not include the physical object 7404,etc.). In some embodiments, the first virtual content completelyreplaces the view of the physical environment in the third view 7410 ofthe three-dimensional environment. In some embodiments, the virtualcontent is displayed overlaying, blocking, or replacing display of atleast a portion of the representation of the physical environment.

In some embodiments, the computer system determines that the secondlocation corresponds to the second type of exercise in accordance with adetermination that the second location has a second type of exerciseequipment (e.g., stairs, steppers, treadmill, etc.) corresponding to thesecond type of exercise. In some embodiments, the computer systemdetermines that the second location corresponds to the second type ofexercise in accordance with a determination that the second location isa location designed for (e.g., having appropriate floor surface,structures, etc. for) the second type of exercise (e.g., hiking,running, etc.).

As shown in FIG. 7P, part (B), the computer system displays a third view7410 of the three-dimensional environment, when the user 7002 has movedto the second location that corresponds to the second type of exercise.In some embodiments, the third view 7410 is an augmented reality viewwith more virtual elements corresponding to a second computer-generatedexperience corresponding to the second location and the second type ofexercise. In some embodiments, the third view 7410 is an augmentedreality view showing a preview or start of a second computer-generatedexperience corresponding to the second location and the second type ofexercise. In some embodiments, the third view 7410 is an augmentedreality view displayed with a higher-level of immersion (e.g.,displaying user interface objects that are part of a second specificapplication experience corresponding to the second type of exercise(e.g., virtual hiking trails, virtual scenery, score boards, exercisestatistics, controls of changing exercise parameters, etc.), that onaggregate occupy a substantial percentage (e.g., greater than 60%,greater than 90%, etc.) of the user's field of view or are displayed ina three-dimensional virtual or augmented reality environment. In thisexample, the virtual content displayed in the third view 7410 of thethree-dimensional environment includes virtual hiking trail 7412 thatreplaced the view of the representations 7004′, 7006′, and/or 7008′ ofvarious portions of the physical environment that are potentially withinthe field of view provided by the display generation component at thesecond location. In some embodiment, all portions of the physicalenvironment in the potential field of view provided by the displaygeneration component are replaced or blocked by the display of thevirtual content. In some embodiments, a portion of the physicalenvironment such as a portion of the user's body, at least a portion ofthe exercise equipment, etc. remain visible in the third view 7410 ofthe three-dimensional environment.

In some embodiments, the computer system determines that the currentlocation corresponds a respective type of exercise in accordance withdetection of a respective type of exercise equipment corresponding tothe respective type of exercise at the current location. In someembodiments, detection of the respective type of exercise equipment isbased on detection of an RFID signal corresponding to the respectivetype of exercise equipment, detection of an image of the respective typeof exercise equipment in a camera feed capturing the current location,detection that the current location matches a registered location forthe respective type of exercise equipment, etc.

In some embodiments, in accordance with a determination that the currentlocation of the user corresponds to a location associated with arespective type of exercise, the computer system displays a view of thethree-dimensional environment that corresponds to the respective type ofexercise, including gradually reducing the visual prominence of therepresentation of the physical environment in the currently displayedview of the three-dimensional environment, while increasing visualprominence of virtual content corresponding to the respective type ofexercise associated with the current location in the view of thethree-dimensional environment. In some embodiments, reducing visualprominence of the representation of the physical environment includesceasing display of more and more portions of the representation of thephysical environment, fading out the representation of the physicalenvironment, etc. In some embodiments gradually increasing a visualprominence of virtual content corresponding to the respective type ofexercise includes starting to display the virtual content, increasingvisibility of the virtual content, increasing a proportion of the fieldof view of the user occupied by the virtual content, increasing anopacity or brightness of the virtual content, etc. in regions of theview of the three-dimensional environment in which the representation ofthe physical environment has been gradually reduced.

In some embodiments, a respective location may correspond to multipletypes of exercises, and the computer system requires that the user makessome movement corresponding to a respective one of the multiple types ofexercises to disambiguate which type of exercise the user wishes toperformance and selects the corresponding virtual content for display inthe view of the three-dimensional environment at the respectivelocation. For example, in some embodiments, the computer system detectsmovement corresponding to a respective one of the multiple types ofexercises associated with the respective location (e.g., starting acharacteristic motion (e.g., starting to walk on a treadmill, steppingon an stair stepper, moving legs back and forth on an elliptical, orstarting rowing on a rowing machine, etc.), stepping onto/sitting downon a piece of exercise equipment corresponding to the respective type ofexercise (e.g., sitting down on a rowing machine, or weight trainingmachine, etc.), getting into a ready posture corresponding to therespective type of exercise (e.g., standing in a ready posture forhitting a virtual tennis ball, sitting down on the floor to startmeditation or yoga, etc.), etc.), and the computer system displays aview of the three-dimensional environment that includes virtual contentcorresponding to the respective type of exercise.

In some embodiments, the computer system gradually changes the virtualcontent that is displayed in the view of the three-dimensionalenvironment in accordance with progress of the respective type exercisesperformed by the user at the respective location. For example, in someembodiments, the view of the real world gradually fades away and/orcease to be displayed, and is gradually replaced by virtual contentcorresponding to the respective type of exercise. In some embodiments,the computer system gradually increases the amount of virtual contentdisplayed in the field of view of the first user until a respectivevirtual environment corresponding to the respective type of exercise isfully displayed via the first display generation component (e.g., thesecond view of the three-dimensional environment includes a virtualenvironment corresponding to the first type of exercise, the third viewof the three-dimensional environment includes a virtual environmentcorresponding to the second type of exercise, etc.). For example, insome embodiments, when an open gym is a location that is associated withboth yoga and dance, after the first user arrives at the open gym, ifthe first user sits down with a Namaste pose, the computer systemdisplays a virtual ocean view with ocean sounds for the user to practiceyoga on a virtual beach; and if the first user stands with a dancer'spose, the computer system displays a virtual stage with dance music forthe user to practice a dance.

In some embodiments, when the computer system detects that the user hasmoved away from a respective location, the computer system ceases todisplay the virtual content corresponding to the type of exerciseassociated with the respective location. For example, in FIG. 7O, if thecomputer system detects that the user 7002 has left the first locationthat includes the physical object 7404; after the view 7408 isdisplayed, the computer system ceases to display the view 7408corresponding to the first type of exercise. In some embodiments, thecomputer system redisplays the view 7405 which does not include thevirtual content that corresponds to either the first type of exercise orthe second type of exercise. In some embodiments, when the computersystem detects that the user has moved from the first location to thesecond location, the computer system displays the virtual content 7410that corresponds to the second type of exercise.

In some embodiments, the computer system displays status information(e.g., progress, duration, speed, force, height, pace, stride length,performance level, scores, number of repetitions completed, etc. duringthe current session, historic statistics, average statistics for thefirst user and/or across multiple users, status of others alsoperforming the same type of exercise, etc.) corresponding to therespective type of exercise when displaying a view of thethree-dimensional environment that includes virtual contentcorresponding to the respective type of exercise.

In some embodiments, the computer system displays health information(e.g., real-time biometric data (e.g., heart rate, blood pressure,breathing rate, body temperature, blood sugar level, etc.), weight, BMI,etc.) corresponding to the user when displaying a view of thethree-dimensional environment that includes virtual contentcorresponding to the respective type of exercise.

In some embodiments, the computer system visually presents progressinformation (e.g., real-time scores, laps completed, laps remaining,duration, number of steps, distance traveled, poses completed, etc.) ofa respective type of exercise that is performed by the user whendisplaying a view of the three-dimensional environment that includesvirtual content corresponding to the respective type of exercise.

In some embodiments, the three-dimensional environment that includes thevirtual content corresponding to a respective type of exercise is animmersive environment, and includes a spatial range that is greater thanthat is included in the currently displayed view of thethree-dimensional environment. For example, as the user turns his/herhead or otherwise change the viewpoint corresponding to the currentlydisplayed view of the three-dimensional environment, different portionsof the virtual content is displayed in the currently displayed view ofthe three-dimensional environment.

In some embodiments, the second and/or third view of thethree-dimensional environment includes a virtual representation of theuser that is shown to perform a respective type of exercise (e.g., basedon previous best records of the first user, based on a presetconfiguration of the first user for the first type of exercise, etc.) incompetition with the user.

In some embodiments, the second and/or third view of thethree-dimensional environment includes a virtual representation of atleast another user different from the user that is shown to perform therespective type of exercise in competition with the user.

As disclosed herein, in some embodiments, the three-dimensionalenvironment that is displayed via the display generation component is avirtual three-dimensional environment that includes virtual objects andcontent at different virtual positions in the three-dimensionalenvironment without a representation of the physical environment. Insome embodiments, the three-dimensional environment is a mixed realityenvironment that displays virtual objects at different virtual positionsin the three-dimensional environment that are constrained by one or morephysical aspects of the physical environment (e.g., positions andorientations of walls, floors, surfaces, direction of gravity, time ofday, etc.). In some embodiments, the three-dimensional environment is anaugmented reality environment that includes a representation of thephysical environment. The representation of the physical environmentincludes respective representations of physical objects and surfaces atdifferent positions in the three-dimensional environment, such that thespatial relationships between the different physical objects andsurfaces in the physical environment are reflected by the spatialrelationships between the representations of the physical objects andsurfaces in the three-dimensional environment. When virtual objects areplaced relative to the positions of the representations of physicalobjects and surfaces in the three-dimensional environment, they appearto have corresponding spatial relationships with the physical objectsand surfaces in the physical environment. In some embodiments, thedisplay generation component includes a pass-through portion in whichthe representation of the physical environment is displayed. In someembodiments, the pass-through portion is a transparent orsemi-transparent (e.g., a see-through) portion of the display generationcomponent revealing at least a portion of physical environmentsurrounding and within the field of view of user. For example, thepass-through portion is a portion of a head-mounted display or heads-updisplay that is made semi-transparent (e.g., less than 50%, 40%, 30%,20%, 15%, 10%, or 5% of opacity) or transparent, such that the user cansee through it to view the real world surrounding the user withoutremoving the head-mounted display or moving away from the heads-updisplay. In some embodiments, the pass-through portion graduallytransitions from semi-transparent or transparent to fully opaque whendisplaying a virtual or mixed reality environment. In some embodiments,the pass-through portion of the display generation component displays alive feed of images or video of at least a portion of physicalenvironment captured by one or more cameras (e.g., rear facing camera(s)of the mobile device or associated with the head-mounted display, orother cameras that feed image data to the electronic device). In someembodiments, the one or more cameras point at a portion of the physicalenvironment that is directly in front of the user's eyes (e.g., behindthe display generation component). In some embodiments, the one or morecameras point at a portion of the physical environment that is notdirectly in front of the user's eyes (e.g., in a different physicalenvironment, or to the side or behind the user). In some embodiments,when displaying virtual objects or content at positions that correspondto locations of one or more physical objects in the physicalenvironment, at least some of the virtual objects are displayed inplaced of (e.g., replacing display of) a portion of the live view (e.g.,a portion of the physical environment captured in the live view) of thecameras. In some embodiments, at least some of the virtual object andcontent are projected onto the physical surfaces or empty space in thephysical environment and are visible through the pass-through portion ofthe display generation component (e.g., viewable as part of the cameraview of the physical environment, or through the transparent orsemi-transparent portion of the display generation component, etc.). Insome embodiments, at least some of the virtual objects and content aredisplayed to overlay a portion of the display and blocks the view of atleast a portion of, but not all of, the physical environment visiblethrough the transparent or semi-transparent portion of the displaygeneration component. In some embodiments, at least some of the virtualobjects are projected directly onto the user's retina at positionsrelative to an image of the representation of the physical environment(e.g., as viewed through a camera view of the physical environment, orthrough a transparent portion of the display generation component,etc.).

In some embodiments, input gestures used in the various examples andembodiments described herein (e.g., with respect to FIGS. 7A-7P, andFIGS. 8-12 ) optionally include discrete, small motion gesturesperformed by movement of the user's finger(s) relative to otherfinger(s) or part(s) of the user's hand, optionally, without requiringmajor movement of the user's whole hand or arm away from their naturallocation(s) and posture(s)) to perform operations immediately prior toor during the gesture) for interacting with a virtual or mixed-realityenvironment, in accordance with some embodiments.

In some embodiments, the input gestures are detected by analyzing dataor signals captured by a sensor system (e.g., sensors 190, FIG. 1 ;image sensors 314, FIG. 3 ). In some embodiments, the sensor systemincludes one or more imaging sensors (e.g., one or more cameras such asmotion RGB cameras, infrared cameras, depth cameras, etc.). For example,the one or more imaging sensors are components of or provide data to acomputer system (e.g., computer system 101 in FIG. 1 (e.g., a portableelectronic device 7100 or an HMD)) that includes a display generationcomponent (e.g., display generation component 120 in FIGS. 1, 3, and 4(e.g., a touch-screen display that serves as a display and atouch-sensitive surface, a stereoscopic display, a display with apass-through portion, etc.). In some embodiments, the one or moreimaging sensors include one or more rear-facing cameras on a side of adevice opposite from a display of the device. In some embodiments, theinput gestures are detected by a sensor system of a head mounted system(e.g., a VR headset that includes a stereoscopic display that provides aleft image for the user's left eye and a right image for the user'sright eye). For example, one or more cameras that are components of thehead mounted system are mounted on the front and/or underside of thehead mounted system. In some embodiments, one or more imaging sensorsare located in a space in which the head mounted system is used (e.g.,arrayed around head mounted system in various locations in a room) suchthat the imaging sensors capture images of the head mounted systemand/or the user of the head mounted system. In some embodiments, theinput gestures are detected by a sensor system of a heads up device(such as a heads up display, automotive windshield with the ability todisplay graphics, window with the ability to display graphics, lens withthe ability to display graphics). For example, one or more imagingsensors are attached to interior surfaces of an automobile. In someembodiments, the sensor system includes one or more depth sensors (e.g.,an array of sensors). For example, the one or more depth sensors includeone or more light-based (e.g., infrared) sensors and/or one or moresound-based (e.g., ultrasonic) sensors. In some embodiments, the sensorsystem includes one or more signal emitters, such as a light emitter(e.g. infrared emitter) and/or sound emitter (e.g., ultrasound emitter).For example, while light (e.g., light from an array of infrared lightemitters having a predetermined pattern) is projected onto a hand (e.g.,hand 7200), an image of the hand under illumination of the light iscaptured by the one or more cameras and the captured image is analyzedto determine a position and/or configuration of the hand. Using signalsfrom image sensors directed to the hand to determine input gestures, asopposed to using signals of touch-sensitive surfaces or other directcontact mechanism or proximity-based mechanisms allow the user to freelychoose whether to execute large motions or remaining relativelystationary when providing the input gestures with his/her hand, withoutexperiencing constraints imposed by a specific input device or inputregion.

In some embodiments, a tap input is, optionally, a tap input of a thumbover index finger (e.g., over a side of the index finger adjacent to thethumb) of a user's hand. In some embodiments, a tap input is detectedwithout requiring lift-off of the thumb from the side of the indexfinger. In some embodiments, a tap input is detected in accordance witha determination that downward movement of the thumb are followed byupward movement of the thumb, with the thumb making contact with theside of the index finger for less than a threshold amount of time. Insome embodiments, a tap-hold input is detected in accordance with adetermination that the thumb moves from the raised position to thetouch-down position and remains in the touch-down position for at leasta first threshold amount of time (e.g., the tap time threshold oranother time threshold that is longer than the tap time threshold). Insome embodiments, the computer system requires that the hand as a wholeremains substantially stationary in location for at least the firstthreshold amount of time in order to detect the tap-hold input by thethumb on the index finger. In some embodiments, the touch-hold input isdetected without requiring that the hand as a whole is keptsubstantially stationary (e.g., the hand as a whole may move while thethumb rests on the side of the index finger). In some embodiments, atap-hold-drag input is detected when the thumb touches down on the sideof the index finger and the hand as a whole moves while the thumb restson the side of the index finger.

In some embodiments, a flick gesture is, optionally, a push or flickinput by a movement of a thumb across index finger (e.g., from the palmside to the back side of the index finger). In some embodiments, theextension movement of the thumb is accompanied by upward movement awayfrom the side of the index finger, e.g., as in an upward flick input bythe thumb. In some embodiments, the index finger moves in the oppositedirection from that of the thumb during the forward and upward movementof the thumb. In some embodiments, a reverse flick input is performed bythe thumb moving from an extended position to a retracted position. Insome embodiments, the index finger moves in the opposite direction fromthat of the thumb during the backward and downward movement of thethumb.

In some embodiments, a swipe gesture is, optionally, a swipe input by amovement of a thumb along index finger (e.g., along a side of the indexfinger adjacent to the thumb or on the side of the palm). In someembodiments, the index finger is optionally in an extended state (e.g.,substantially straight) or a curled up state. In some embodiments, theindex finger moves between the extended state and the curled up stateduring the movement of the thumb in a swipe input gesture.

In some embodiments, different phalanges of various fingers correspondto different inputs. A tap input of thumb over various phalanges ofvarious fingers (e.g., index finger, middle finger, ring finger, and,optionally, pinky finger) are optionally mapped to different operations.Similarly, in some embodiments, different push or click inputs can beperformed by the thumb across different fingers and/or different partsof a finger to trigger different operations in a respective userinterface contact. Similarly, in some embodiments, different swipeinputs performed by the thumb along different fingers and/or indifferent directions (e.g., toward the distal or proximal end of afinger) trigger different operations in a respective user interfacecontext.

In some embodiments, the computer system treats tap inputs, flickinputs, and swipe inputs are treated as different types of inputs basedon movement types of the thumb. In some embodiments, the computer-systemtreats inputs having different finger locations that are tapped,touched, or swiped by the thumb as different sub-input-types (e.g.,proximal, middle, distal subtypes, or index, middle, ring, or pinkysubtypes) of a given input type (e.g., a tap input type, a flick inputtype, a swipe input type, etc.). In some embodiments, the amount ofmovement performed by the moving finger (e.g., thumb) and or othermovement metrics associated with the movement of the finger (e.g.,speed, initial speed, ending speed, duration, direction, movementpattern, etc.) is used to quantitatively affect the operation that istriggered by the finger input.

In some embodiments, the computer-system recognizes combination inputtypes that combines a sequence of movements by the thumb, such as atap-swipe input (e.g., touch-down of thumb on a finger followed byswiping along the side of the finger), a tap-flick input (e.g.,touch-down of thumb over a finger followed by a flick across the fingerfrom palm side to back side of the finger), a double tap input (e.g.,two consecutive taps on the side of a finger at about the samelocation), etc.

In some embodiments, the gesture inputs are performed by an index fingerinstead of the thumb (e.g., index finger performs the tap or swipe onthe thumb, or the thumb and the index finger move toward each other toperform a pinch gesture, etc.). In some embodiments, a wrist movement(e.g., a flick of the wrist in a horizontal direction, or a verticaldirection) is performed immediately preceding, immediately succeeding(e.g., within a threshold amount of time) or contemporaneously with thefinger movement inputs to trigger additional operations, differentoperations, or modified operations in the current user interfacecontext, as compared to the finger movement inputs without the modifierinput by the wrist movement. In some embodiments, the finger inputgestures performed with the user's palm facing the user's face aretreated as a different type of gestures from finger input gesturesperformed with the user's palm facing away from the user's face. Forexample, a tap gesture performed with the user's palm facing the userperforms an operation with added (or reduced) privacy safeguard ascompared to an operation (e.g., the same operation) performed inresponse to a tap gesture performed with the user's palm facing awayfrom the user's face.

Although one type of finger input may be used to trigger a type ofoperation in the examples provided in this disclosure, other types offinger input are optionally used for trigger the same type of operationin other embodiments.

Additional descriptions regarding FIGS. 7A-7P are provided below inreferences to methods 8000, 9000, 10000, 11000, and 12000 described withrespect to FIGS. 8-12 below.

FIG. 8 is a flowchart of a method of supporting interaction with a userinterface object in a computer-generated three-dimensional environmentthat is shared between two or more users, in accordance with someembodiments.

In some embodiments, the method 8000 is performed at a computer system(e.g., a first computer system 101 in FIG. 1 ) including a displaygeneration component (e.g., display generation component 120 in FIGS. 1,3, and 4 ) (e.g., a heads-up display, a display, a touchscreen, aprojector, etc.) and one or more cameras (e.g., a camera (e.g., colorsensors, infrared sensors, and other depth-sensing cameras) that pointsdownward at a user's hand or a camera that points forward from theuser's head). In some embodiments, the method 8000 is governed byinstructions that are stored in a non-transitory computer-readablestorage medium and that are executed by one or more processors of acomputer system, such as the one or more processors 202 of computersystem 101 (e.g., control unit 110 in FIG. 1A). Some operations inmethod 8000 are, optionally, combined and/or the order of someoperations is, optionally, changed.

In some embodiments, the method 8000 is performed at a computer system(e.g., first computer system 101 in FIG. 1 ) that is in communicationwith a display generation component (e.g., display generation component120 in FIGS. 1, 3, and 4 , display generation component 7100, etc.)(e.g., a heads-up display, an HMD, a display, a touchscreen, aprojector, etc.) and one or more input devices (e.g., cameras,controllers, touch-sensitive surfaces, joysticks, buttons, etc.). Insome embodiments, the computer system is an integrated device with oneor more processors and memory enclosed in the same housing as thedisplay generation component and at least some of the one or more inputdevices. In some embodiments, the computer system includes a computingcomponent that includes one or more processors and memory that isseparate from the display generation component and/or the one or moreinput devices. In some embodiments, the display generation component andthe one or more input devices are integrated and enclosed in the samehousing.

In the method 8000, the computer system displays (8002) a first userinterface object (e.g., user interface object 7016 in FIG. 7B, anotheruser interface object, etc.) (e.g., a representation of an application,a user interface that includes a plurality of user interface objects(e.g., selectable avatars, selectable menu items, selectable devicecontrols, selectable content items, slider controls, buttons, etc.), avirtual three-dimensional object, a control, a control panel thatincludes multiple controls corresponding to different functions oroperations, an information item, a media item, etc.) in a first view ofa three-dimensional environment (e.g., first view 7015-1 in FIG. 7B,another first view, etc.), wherein the three-dimensional environment isat least partially shared (e.g., at least a spatial portion of theenvironment is shared, the environment is shared during at least aperiod of time, objects in the environment are shared fully or partially(e.g., simultaneously viewable and accessible, simultaneously viewablebut not simultaneously accessible, viewable but not accessible whenothers have control (e.g., said others can be viewing or not viewing theobject), etc.) between a first user (e.g., user 7102 in FIGS. 7A-7C) anda second user (e.g., user 7002 in FIGS. 7A-7C) (e.g., when at least aportion of the three-dimensional environment (e.g., the portion shown inthe first view of the three-dimensional environment, another portion ofthe three-dimensional environment, etc.) is displayed for viewing byboth the first user and the second user at the same time, and/or whensome or all of the virtual objects (e.g., including the first userinterface object, another user interface object, etc.) in thethree-dimensional environment are concurrently displayed in thethree-dimensional environment shown to both the first user and thesecond user, etc.), wherein the first user interface object is displayedwith a first set of appearance properties (e.g., as shown in FIG. 7B)(e.g., the normal appearance (e.g., first shape, first size, firstcolor, first opacity, first level of saturation, first level ofluminance, etc.) of the first user interface object as displayed by thesecond display generation component to the second user) at a firstposition in the first view of the three-dimensional environment (e.g.,first view 7015-1 in FIG. 7B). While displaying the first user interfaceobject with the first set of appearance properties at the first positionin the first view of the three-dimensional environment, the computersystem detects (8004) a first user input provided by the first user,wherein the first user input is directed to the first user interfaceobject (e.g., detecting the user input includes detecting movement of aportion of the first user to a first location in the physicalenvironment, where the first location in the physical environmentcorresponds a respective position of the first user interface object inthe first view of the three-dimensional environment; detecting the userinput includes detecting a gaze input directed to the first userinterface object and a control input (e.g., a finger movement gesture,an in air gesture, an input provided by a controller, etc.) that isdetected in conjunction with the gaze input; etc.). In response todetecting (8006) the first user input that is directed to the first userinterface object and in accordance with a determination that the seconduser (e.g., user 7002 in FIGS. 7A-7C) is not currently interacting withthe first user interface object (e.g., user interface object 7016 inFIGS. 7A-7C) (e.g., the first user interface object does not have apreset spatial relationship to virtual position of the second user inthe first view of the three-dimensional environment (e.g., the firstuser interface object is in not inside the representation of the seconduser's palm or hand, the first user interface object is outside of thesecond user's private space that is visible within the first view of thethree-dimensional environment, etc.), the second user is notcontrolling, selecting, moving, modifying, and/or otherwise interactingwith the first user interface object through a second computer systemthat displays a second view of the three-dimensional environment in theat least partially shared three-dimensional environment, etc.), thecomputer system performs (8008) a first operation with respect to thefirst user interface object in accordance with the first user input(e.g., showing the first user interface object being grabbed or moved bythe first user in accordance with the first user input (e.g., movedtoward the user, moved in accordance with the movement of the first userinput, etc.), showing a ghost image of the first user interface objectbeing grabbed and/or moving into a representation of the first user'shands, etc.). In some embodiments, the first user interface objectcontinues to be displayed with the first set of appearance properties(e.g., at its original location or in a representation of the firstuser's hand, etc.) in accordance with a determination that the seconduser was not interacting with the first user interface object when thefirst user input was detected. In response to detecting (8006) the firstuser input that is directed to the first user interface object (e.g.,the user interface object 7016 in FIGS. 7A-7C) and in accordance with adetermination that the second user is currently interacting with thefirst user interface object (e.g., the first user interface object hasthe preset spatial relationship to a virtual position of the second userin the first view of the three-dimensional environment (e.g., the firstuser interface object is in the representation of the second user's palmor hand, the first user interface object is within the second user'sprivate space that is visible within the first view of thethree-dimensional environment, etc.), the second user is controlling,selecting, moving, modifying, and/or otherwise interacting with thefirst user interface object through the second computer system thatdisplays a second view of the three-dimensional environment in theshared three-dimensional environment, etc.), the computer systemdisplays (8010) a visual indication that the first user interface objectis not available for interaction with the first user, wherein displayingthe visual indication includes changing at least one of an appearance ofthe first user interface object or a position of the first userinterface object in the first view of the three-dimensional environment(e.g., in FIG. 7C, the appearance of the user interface object 7016 ischanged in the first view 7015-1 shown to the first user 7102). In someembodiments, the computer system displays the first user interfaceobject with a second set of appearance properties (e.g., second shape,second size, second color, second opacity, second level of saturation,second level of luminance, etc.) that are different from the first setof appearance properties (e.g., the second set of appearance propertiesprovide a visual indication that the first user interface object is incontrol of the second user at this moment, and is not available forinteracting with the first user), and/or moves the first user interfaceobject out of the way when the first user tries to grab it. In someembodiments, the first user interface object maintains its appearanceand/or position in the view of the at least partially sharedthree-dimensional environment displayed to the second user, as thevisual indication only needs to be displayed to the first user. In someembodiments, the visual indication is displayed while the second user isinteraction with the first user interface object in the at leastpartially shared three-dimensional environment. The computer systemforgoes (8014) performing the first operation with respect to the firstuser interface object in accordance with the first user input. In someembodiments, the computer system does not show the first user interfaceobject being grabbed by the representation of the first user or does notshow the first user interface object moving in accordance with themovement of the first user input (e.g., object is not moving to avoidbeing grabbed by the first user's hand, object is not shrinking orchanging shape to avoid being grabbed by the representation of the firstuser, etc.). In some embodiments, the computer system does not show aghost image of the first user interface object moving into therepresentation of the first user's hand.

These features are illustrated, for example, in FIGS. 7A-7C, where thefirst user 7102 and the second user 7002 shares the three-dimensionalenvironment shown respectively via the display generation components7200 and 7100. When the second user 7002 has control of the first userinterface object 7016 (e.g., is interacting with the first userinterface object 7016, holds the first user interface object 7016 or arepresentation thereof via the representation 7028″ of the hand 7028 inthe second view 7015-2 (also representation 7028′ in the first view7015-1 shown to the first user 7102), etc.), if the first user 7102makes an attempt to interact with the first user interface object 7016through a movement of the first user's hand 7102, the computer system ofthe first user 7102 changes an appearance of the first user interfaceobject 7016 in the first view 7015-1 shown via the first displaygeneration component 7200, and does not perform an operationcorresponding to the first user interface object 7016. In contrast, ifthe second user 7002 is not interacting with the first user interfaceobject 7016, then the computer system performs the first operation inaccordance with the movement of the first user's hand 7202. This isindirectly illustrated by the interaction between the second user 7002and the first user interface object 7016 in FIG. 7B, where the firstuser 7102 does not have control or is not interacting with the firstuser interface object 7016 (e.g., consider reversal of the roles of thefirst user and the second user in that scenario).

In some embodiments, the computer system changes the appearance of thefirst user interface object as the visual indication that the first userinterface object is not available for interaction with the first user,and changing the appearance of the first user interface object includeschanging at least one of the first set of appearance properties of thefirst user interface object (e.g., increasing a transparency level,reducing color saturation, reducing opacity, blurring, darkening,reducing resolution, shrinking in size, etc. of the first user interfaceobject, optionally, while maintaining appearance of the surroundingenvironment of the first user interface object (e.g., not changing thevisual prominence of the surrounding environment)) to reduce visualprominence of the first user interface object (e.g., in FIG. 7C, theappearance of the user interface object 7016 is changed in the firstview 7015-1 shown to the first user 7102). In some embodiments, thecompute system changes the appearance of the first user interface objectwhile maintaining a position of the first user interface object in thefirst view of the three-dimensional environment that is determinedindependent of the first user input (e.g., the first position, anotherposition determined in response to the interaction between the firstuser interface object and the second user, another position determinedin accordance with preprogramed autonomous movement of the first userinterface object (e.g., the first user interface object has a presetmovement pattern or preset animated effect, etc.), another positiondetermined in accordance with other events in the computer system,etc.). In some embodiments, in response to detecting the first userinput that is directed to the first user interface object, in accordancewith a determination that the second user is not currently interactingwith the first user interface object, the computer system does notchange the appearance of the first user interface object to reducevisual prominence of the first user interface object, and the computersystem performs the first operation with respect to the first userinterface object in accordance with the first user input (e.g., theappearance and visual prominence of the first user interface object ismaintained, or the appearance may be changed as a result of performingthe first operation but not with a goal to reduce visual prominence ofthe first user interface object, etc.).

Changing the appearance of the first user interface object, includingchanging at least one of the first set of appearance properties of thefirst user interface object to reduce visual prominence to the firstuser interface object, as a visual indication that the first userinterface object is not available for interaction with the first user,provides improved visual feedback to the users (e.g., improved visualfeedback that the first user interface object is not available forinteraction with the first user). Providing improved feedback enhancesthe operability of the device, which, additionally, reduces power usageand improves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the computer system detects termination of thefirst user input that is directed to the first user interface object(e.g., detecting movement of a portion of the first user away from thefirst location in the physical environment that corresponds to therespective position of the first user interface object in the first viewof the three-dimensional environment, detecting the gaze input that wasdirected to the first user interface object moving away from the firstuser interface object, detecting the hand of the first user thatprovided the first user input moving out of the posture required tomaintain the first user input, etc.). In response to detecting thetermination of the first user input that is directed to the first userinterface object, the computer system restores (e.g., to the levelexisted immediately prior to detecting the first user input, or prior tochanges being made in response to detecting the first user input, etc.)at least one of the first set of appearance properties of the first userinterface object that was changed in response to the first user input,to restore the visual prominence of the first user interface object. Insome embodiments, the computer system restores the increasedtransparency level, restores the decreased color saturation, restoresthe reduced opacity, ceases to blur and/or darken, restores the reducedresolution, restores the reduced size, etc. of the first user interfaceobject, optionally, while maintaining appearance of the surroundingenvironment of the first user interface object (e.g., not changing thevisual prominence of the surrounding environment). For example, in someembodiments, when the first user reaches out his/her hand toward alocation that corresponds to a virtual object with which the second useris currently interacting, the virtual object appears to fade out orbecome dimmer when the first user's hand is at a location in thephysical environment that corresponds to the position of the virtualobject in the three-dimensional environment. When the first user thensubsequently moves his/her hand away from that location, the appearanceof the virtual object is restored (e.g., no longer appearing to be fadedout or dim). This is illustrated in FIG. 7B (following FIG. 7C), whereif the first user 7102 ceases to attempt to interact with the first userinterface object 7016, the appearance of the first user interface object7016 is no longer altered in the first view 7015-1 shown to the firstuser 7102.

Restoring at least one of the first set of appearance properties of thefirst user interface object that was changed in response to the firstuser input, to restore visual prominence of the first user interfaceobject, in response to detecting the termination of the first user inputthat is directed to the first user interface object, provides improvedvisual feedback to the users (e.g., improved visual feedback that thefirst user interface object is available for interaction). Providingimproved feedback enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, while continuing to detect the first user input(e.g., detecting the portion of the first user remaining at the firstlocation in the physical environment that corresponds to the respectiveposition of the first user interface object in the first view of thethree-dimensional environment at a time when the first user input wasinitially detected, detecting the gaze input that was directed to thefirst user interface object remaining at the same position in thethree-dimensional environment, detecting the hand of the first user thatprovided the first user input remaining in the required posture formaintain the first user input, etc.), the computer system detectsmovement of the first user interface object away from the first positionin the first view of the three-dimensional environment independent ofthe detection of the first user input (e.g., in accordance with a userinput provided by the second user, in accordance with intrinsic movementpattern of the first user interface object, in response to other eventsin the computer system that is independent of the first user input,etc.). In response to detecting the movement of the first user interfaceobject away from the first position in the first view of thethree-dimensional environment independent of the detection of the firstuser input, the computer system restores (e.g., to the level existedimmediately prior to detecting the first user input, or prior to changesbeing made in response to detecting the first user input, etc.) at leastone of the first set of appearance properties of the first userinterface object that was changed in response to the first user input,to restore the visual prominence of the first user interface object. Insome embodiments, the computer system restores the increasedtransparency level, restores the decreased color saturation, restoresthe reduced opacity, ceases to blur and/or darken, restores the reducedresolution, restores the reduced size, etc. of the first user interfaceobject, optionally, while maintaining appearance of the surroundingenvironment of the first user interface object (e.g., not changing thevisual prominence of the surrounding environment). For example, in someembodiments, when the first user reaches out his/her hand toward alocation that corresponds to a virtual object with which the second useris currently interacting, the virtual object appears to fade out orbecome dimmer when the first user's hand is at a location in thephysical environment that corresponds to the position of the virtualobject in the three-dimensional environment. When the first userinterface object is then subsequently moved away (e.g., moved by thesecond user, according to its own movement pattern, according to othersystem-generated events unrelated to the first user input, etc.) fromits current position and away from the position that corresponds to thecurrent location of the first user's hand, the appearance of the virtualobject is restored (e.g., no longer appearing to be faded out or dim).

Restoring at least one of the first set of appearance properties of thefirst user interface object that was changed in response to the firstuser input, to restore the visual prominence of the first user interfaceobject, in response to detecting the movement of the first userinterface object away from the first position in the first view of thethree-dimensional environment independent of the detection of the firstuser input, provides improved visual feedback to the users (e.g.,improved visual feedback that the first user interface object has beenmoved away from the first position). Providing improved feedbackenhances the operability of the device, which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, displaying the visual indication that the firstuser interface object is not available for interaction with the firstuser includes maintaining changes to the appearance of the first userinterface object made in response to the first user input until thesecond user ceases to interact with the first user interface object. Forexample, the changed appearance of the user interface object 7016 willbe maintained even after the first user 7102 ceases his attempt tointeract with the first user interface object 7016, until the seconduser 7002 no longer controls the first user interface object 7016 inexclusion of the first user 7102. For example, in some embodiments, oncethe visual indication is displayed in response to detecting the firstuser input and in accordance with the determination that the second userwas interacting with the first user interface object at the time thatthe first user input is initially detected, the computer systemcontinues to display the visual indication (e.g., the changed appearanceof the first user interface object, changed position, etc.) inaccordance with a determination that the second user is stillinteracting with the first user interface object (e.g., the second usercontinues to keep the virtual object at a position that corresponds tothe location of the second user's palm or hand, and/or continues toselect, modify, or otherwise interact with the virtual object throughthe operation of the computer system of the second user, etc.).

In some embodiments, the visual indication continues to be displayedeven when the computer system detects that the first user input has beenterminated and that the first user is not currently providing any inputto attempt to interact with the first user interface object. In someembodiments, the visual indication is maintained for a period of time,irrespective of whether the first user input is maintained or if thefirst user continues to attempt to interact with the first userinterface object, but not necessarily until the second user has stoppedinteracting with the first user interface object. In some embodiments,the computer system of the first user determines that the second user isno longer interacting with the first user interface object, and inresponse to detecting that the second user is no longer interacting withthe first user interface object, the computer system ceases to displaythe visual indication that the first user interface object is noavailable for interaction with the first user (e.g., the computer systemceases to display the first user interface object in the faded or dimmedstate, and restores the original appearance properties of the first userinterface object that had been changed in response to the detection ofthe first user input). In some embodiments, the persistent visualindication that the first user interface object is still within thecontrol of the second user and/or is not available for interaction withthe first user helps the first user to know when the device is ready torespond to another attempt to interact with the first user interfaceobject and avoid repeated failures in trying to interact with the firstuser interface object.

Maintaining changes to the appearance of the first user interface objectmade in response to the first user input until the second user ceases tointeract with the first user interface object provides improved visualfeedback to the users (e.g., improved visual feedback that the seconduser is interacting with the first user interface object, improvedvisual feedback that the first user interface object is not availablefor interaction with the first user, etc.). Providing improved feedbackenhances the operability of the device, which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, in response to detecting the first user input thatis directed to the first user interface object (e.g., user interfaceobject 7016 in FIGS. 7B-7C) and in accordance with a determination thatthe second user (e.g., user 7002 in FIGS. 7B-7C) is not currentlyinteracting with the first user interface object (e.g., the first userinterface object does not have a preset spatial relationship to avirtual position of the second user in the first view of thethree-dimensional environment (e.g., the first user interface object isin not inside the representation of the second user's palm or hand, thefirst user interface object is outside of the second user's privatespace that is visible within the first view of the three-dimensionalenvironment, etc.), the second user is not controlling, selecting,moving, modifying, and/or otherwise interacting with the first userinterface object through second computer system that displays a secondview of the three-dimensional environment in the at least partiallyshared three-dimensional environment, etc.), the computer systemdisplays the first user interface object at a second position in thefirst view of the three-dimensional environment, wherein the secondposition is selected in accordance with a current location of a hand ofthe first user (e.g., user 7102 in FIGS. 7B-7C, or another user, etc.).In some embodiments, the second position is selected to correspond tothe current location of the first user's hand (e.g., hand 7202 in FIGS.7B-7C, or another hand, etc.) providing the first user input (e.g.,overlays, replaces display of, modifying the appearance of, etc. therepresentation of the first user's hand), or the second position isselected to correspond to a position in the three-dimensionalenvironment or a location in the physical environment that is pointed toor indicated by the first user input, etc.

In some embodiments, the computer system shows the first user interfaceobject being grabbed by the representation of the hand of the first userin the first view of the three-dimensional environment, shows the firstuser interface object moving in a direction that corresponds to amovement direction of the first user input provided by the first user'shand (e.g., upward movement of the user's hand causes the virtual objectto be lifted up from a representation of a table top, upward movement ofthe user's hand causes the virtual object to jump up from therepresentation of a table top and land in the representation of theuser's hand, etc.), shows the first user interface object (e.g., userinterface object 7016 in FIGS. 7B-7C, or another user interface object,etc.) moving to a position indicated by the first user's hand (e.g. aflick and point gesture by the first user's hand causes the virtualobject to move up from its original position and move to the positionindicated by the pointing finger of the first user's hand, etc.), etc.In response to detecting the first user input that is directed to thefirst user interface object and while displaying the first userinterface object at the second position that is selected in accordancewith the current location of the hand of the first user (e.g., while thefirst user interface object is displayed at the position thatcorresponds to the representation of the first user's hand (e.g.,appears to be held by the first user's hand), while the first userinterface object is displayed hovering at a position that is selected bythe first user input, etc.), the computer system detects movement of thehand of the first user (e.g., the user 7102, or another user, etc.) thatcorresponds to a throwing gesture of the hand of the first user. In someembodiments, the computer system detects the first user's hand movingfrom a location close to the first user to another location farther awayfrom the first user, and, optionally toward a location that correspondsto a position of the representation of the second user. In someembodiments, the computer system detects the first user flicking anindex finger away from the first user, optionally toward a location thatcorresponds to a position of the representation of the second user,and/or detecting the first user performing tossing motion or throwingmotion using his/her hand. In some embodiments, detecting the throwinggesture includes detecting a quick acceleration of the hand followed bya quick deceleration. In some embodiments, detecting the throwinggesture includes detecting a closed hand opening (and optionally, inconjunction with detecting the quick deceleration). In response todetecting the first user input that is directed to the first userinterface object and in response to detecting the movement of the handof the first user that corresponds to the throwing gesture of the handof the first user, the computer system moves the first user interfaceobject in the first view of the three-dimensional environment in a firstdirection that corresponds to a direction of the movement of the hand ofthe first user and rotating the first user interface object duringmovement of the first user interface object. In some embodiments, as thefirst user interface object moves in the direction that corresponds tothe direction of the throwing gesture, the first user interface objectalso rotates around a virtual center of weight (e.g., a geometriccenter, another point in or out of the first user interface objectdepending on the object simulated by the first user interface object,etc.) of the first user interface object (e.g., to simulate conservationof angular momentum during the linear motion of the first user interfaceobject, to simulate a physical effect of the throwing gesture on thefirst user interface object, to show a predefined user-facing side ofthe first user interface object toward the second user at thedestination end of the throwing gesture, to land on a representation ofa physical surface or on a virtual surface with a predefined uprightorientation, etc.).

In some embodiments, when the second user has been interacting with thefirst user interface object and subsequently performs the throwinggesture to throw away the first user interface object in thethree-dimensional environment, the computer system shows the first userinterface object moving in the first view of the three-dimensionalenvironment in a second direction that corresponds to a direction of themovement of the hand of the second user and rotating the first userinterface object during movement of the first user interface object(e.g., as the first user interface object moves in the direction of thethrowing gesture, the first user interface object also rotates around avirtual center of weight of the first user interface object (e.g., tosimulate conservation of angular momentum during the linear motion ofthe first user interface object, to simulate a physical effect of thethrowing gesture on the first user interface object, to show apredefined user-facing side of the first user interface object towardthe first user at the destination end of the throwing gesture, to landon a representation of a physical surface or on a virtual surface with apredefined upright orientation, etc.)).

Moving the first user interface object in the first view of thethree-dimensional environment in a first direction that corresponds to adirection of the movement of the hand of the first user, and rotatingthe first user interface object during movement of the first userinterface object, in response to detecting the movement of the hand ofthe first user that corresponds to the throwing gesture of the hand ofthe first user, provides improved visual feedback to the users (e.g.,improved visual feedback that the computer system has detected thethrowing gesture of the hand of the first user, improved visual feedbackthat the first user interface object is being moved, etc.). Providingimproved feedback enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, rotating the first user interface object (e.g.,user interface object 7016, or another user interface object, etc.)during movement of the first user interface object includes: inaccordance with a determination that the direction of the movement ofthe hand of the first user points toward a representation of the seconduser in the first view of the three-dimensional environment, rotatingthe first user interface object in a first manner (e.g., rotate thefirst user interface object by a first amount in a first rotationaldirection, by a second amount in a second rotational direction, and/orby a third amount in a third rotational direction, etc.) such that thefirst user interface object has a first preset orientation in thethree-dimensional environment when arriving at a destination position inthe three-dimensional environment (e.g., the position of therepresentation of the second user's hand, a representation of a surfaceassociated with the second user, etc.) that is selected in accordancewith the movement of the hand of the first user in the physicalenvironment. In some embodiments, the first preset orientation isdifferent from the orientation that the first user interface object hadwhen the first user interface object started the movement in response tothe throwing gesture of the first user. In some embodiments, the firstpreset orientation is an orientation in which a preset front-facing sideof the first user interface object faces toward the representation ofthe second user. In some embodiments, the first preset orientation is anorientation in which the first user interface object is upright whencaught by and/or when resting on the representation of the hand of thesecond user, etc. In some embodiments, when the second user has beeninteracting with the first user interface object and subsequentlyperforms the throwing gesture in the direction that corresponds to thedirection of the representation of the first user, the computer system,in accordance with a determination that the direction of the movement ofthe hand of the second user points toward the viewpoint of the firstview of the three-dimensional environment, rotates the first userinterface object (e.g., by a first amount in a first rotationaldirection, by a second amount in a second rotational direction, and/orby a third amount in a third rotational direction, etc.) in a respectivemanner such that the first user interface object has the first presetorientation (e.g., the first preset orientation is different from theorientation that the first user interface object had when the first userinterface object started the movement in response to the throwinggesture of the second user, the first preset orientation is anorientation in which a preset front-facing side of the first userinterface object faces toward a representation of the first user, or thefirst preset orientation is an orientation in which the first userinterface object is upright when caught by and/or when resting on therepresentation of the hand of the first user, etc.) relative to thethree-dimensional environment when arriving at a destination position inthe three-dimensional environment that is selected in accordance withthe movement of the hand of the second user in the physical environment.

Rotating the first user interface object in a first manner such that thefirst user interface has a first preset orientation in thethree-dimensional environment when arriving at a destination position inthe three-dimensional environment that is selected in accordance withthe movement of the hand of the first user in the physical environment,reduces the number of inputs to display the first user interface objectwith the desired orientation at the destination position (e.g., theusers do not need to perform additional gestures to rotate the firstuser interface object to the desired orientation (e.g., for viewing)after the first user interface object is rotated during the movement ofthe first user interface object). Reducing the number of inputs neededto perform an operation enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, rotating the first user interface object duringmovement of the first user interface object includes: in accordance witha determination that the direction of the movement of the hand of thefirst user (e.g., first user 7102 in FIGS. 7A-7C, another user, etc.)points toward a representation of a first surface (e.g., a physicalsurface (e.g., a wall, a tabletop, the seat of a couch, etc.), a virtualsurface (e.g., a virtual shelf, a virtual wall, a virtual couch, etc.),etc.) in the first view (e.g., first view 7015-1 in FIGS. 7B-7C, anotherview, etc.) of the three-dimensional environment, rotating the firstuser interface object (e.g., first user interface object 7016 in FIGS.7B-7C, or another user interface object, etc.) in a second manner (e.g.,rotate the first user interface object by a first amount in a firstrotational direction, by a second amount in a second rotationaldirection, and/or by a third amount in a third rotational direction,etc.) such that the first user interface object has a second presetorientation relative to the representation of the first surface in thethree-dimensional environment when arriving at a destination position onthe representation of the first surface that is selected in accordancewith the movement of the hand of the first user in the physicalenvironment. In some embodiments, the second preset orientation isdifferent from the orientation that the first user interface object hadwhen the first user interface object started the movement in response tothe throwing gesture of the first user. In some embodiments, the secondpreset orientation is an orientation in which a preset front-facing sideof the first user interface object faces toward a representation of thefirst user. In some embodiments, the second preset orientation is anorientation in which the first user interface object is upright whenlanding on the representation of the first surface, etc.

In some embodiments, when the second user has been interacting with thefirst user interface object and subsequently performs the throwinggesture in the direction that corresponds to the direction of therepresentation of the first surface, the computer system, in accordancewith a determination that the direction of the movement of the hand ofthe second user points toward the representation of the first surface inthe three-dimensional environment, rotates the first user interfaceobject (e.g., by a first amount in a first rotational direction, by asecond amount in a second rotational direction, and/or by a third amountin a third rotational direction, etc.) in a respective manner such thatthe first user interface object has the second preset orientationrelative to the representation of the first surface in thethree-dimensional environment when arriving at a destination position onthe representation of the first surface that is selected in accordancewith the movement of the hand of the second user in the physicalenvironment. In some embodiments, irrespective of which user made thethrowing gesture in the direction of the representation of the firstsurface, the first user interface object is rotated during its movementtoward the representation of the first surface in a respective mannersuch that the first user interface object lands on the representation ofthe first surface with a preset spatial relationship (e.g., orientation,location, etc.) relative to the representation of the first surface. Forexample, in some embodiments, when the first user interface object is avirtual picture frame, the virtual picture frame rotates while beingthrown toward the representation of a table, and lands on the table withan upright orientation facing the user that performed the throwinggesture. In some embodiments, when the virtual picture frame is throwntoward the representation of a wall, the virtual picture frame rotatesduring movement toward the representation of the wall and lands on therepresentation of the wall with its back parallel to the representationof the wall. In some embodiments, when the virtual picture frame isthrown toward the second user by the first user, the virtual pictureframe rotates to have its front side face toward the representation ofthe second user when the virtual picture frame lands on therepresentation of the palm of the second user.

Rotating the first user interface object in a second manner such thatthe first user interface object has a second preset orientation relativeto the representation of the first surface in the three-dimensionalenvironment when arriving at a destination position on therepresentation of the first surface that is selected in accordance withthe movement of the hand of the first user in the physical environment,reduces the number of inputs needed to display the first user interfaceobject with the desired orientation on the first surface (e.g., theusers do not need to perform additional gestures to rotate the firstuser interface object to the desired orientation (e.g., for viewing)after the first user interface object is rotated during the movement ofthe first user interface object). Reducing the number of inputs neededto perform an operation enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, the computer system changes the position of thefirst user interface object (e.g., user interface object 7016 in FIGS.7B-7C, or another user interface object, etc.) in the first view (e.g.,7015-1 in FIG. 7C, or another view, etc.) of the three-dimensionalenvironment as the visual indication that the first user interfaceobject is not available for interaction with the first user, andchanging the position of the first user interface object in the firstview of the three-dimensional environment includes moving the first userinterface object from the first position to maintain at least a presetdistance between the first user interface object and a representation ofa hand of the first user that provided the first user input (e.g., thefirst user interface object appears to move in one or more directions toavoid the representation of the hand of the first user that tries tograb the first user interface object). In some embodiments, the movementof the first user interface object is accompanied by changes made to theappearance of the first user interface object (e.g., the first userinterface object appears to be faded or dimmed while moving to avoid therepresentation of the hand of the first user getting too close toitself). In some embodiments, in response to detecting the first userinput that is directed to the first user interface object, in accordancewith a determination that the second user is not currently interactingwith the first user interface object, the computer system does not movethe first user interface object from the first position until therepresentation of the first user's hand reaches the first position, andthe computer system then moves the first user interface object away fromthe first position or manipulates the first user interface object inaccordance with the subsequent movement of the hand from the firstposition (e.g., the subsequent movement of the first user interfaceobject is not to avoid the hand, but to move with the hand or to followthe hand). In some embodiments, in response to detecting the first userinput that is directed to the first user interface object, in accordancewith a determination that the second user is not currently interactingwith the first user interface object, the computer system moves thefirst user interface object from the first position toward therepresentation of the first user's hand until the positions of the firstuser interface object and the representation of the hand overlap, andthe computer system then moves the first user interface object ormanipulates the first user interface object in accordance with thesubsequent movement of the hand (e.g., the first user interface objectmoves with the representation of the hand or follows the representationof the hand).

Moving the first user interface object from the first position tomaintain at least a preset distance between the first user interfaceobject and a representation of a hand of the first user that providedthe first user input, as the visual indication that the first userinterface object is not available for interaction with the first user,provides improved visual feedback to the users (e.g., improved visualfeedback that the first user interface object is not available forinteraction with the first user). Providing improved feedback enhancesthe operability of the device, which, additionally, reduces power usageand improves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, performing the first operation with respect to thefirst user interface object (e.g., user interface object 7016 in FIGS.7B-7C, or another user interface object, etc.) in accordance with thefirst user input includes moving the first user interface object towarda representation of a hand of the first user (e.g., the hand 7202 inFIGS. 7B-7C, or another hand, etc.) (e.g., the hand that is providingthe first user input, a hand of the first user that is different fromthe hand that provided the first user input, either hand of the firstuser, etc.). In some embodiments, the computer system shows the firstuser interface object moving toward the representation of the hand asthe hand moves toward the first user interface object to indicate thatthe first user interface object is available to be grabbed by the firstuser's hand. In some embodiments, the movement of the first userinterface object stops when the position of the first user interfaceobject overlaps with the position of the representation of the hand, andthe computer system then moves the first user interface object ormanipulates the first user interface object in accordance with thesubsequent movement of the hand, if any. In some embodiments, themovement of the first user interface object is only for a limiteddistance away from the first position (e.g., the first user interfaceobject moves toward the representation of the first user's hand a littlebit when the representation of the first user's hand approaches thefirst user interface object to within a threshold distance of the firstposition), and the movement provides visual feedback to the first userthat the first user interface object is available for interaction withthe first user (e.g., when the first user provides the correct selectiongesture, in response to the representation of the first user's handmoving closer and grabbing the first user interface object, etc.).

Moving the first user interface object toward a representation of a handof the first user in accordance with a determination that the seconduser is not currently interacting with the first user interface object,provides improved visual feedback to the users (e.g., improved visualfeedback that the user interface object is available for interactionwith the first user, improved visual feedback that the computer systemis performing the operation with respective to the first user interfaceobject, etc.). Providing improved feedback enhances the operability ofthe device, which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, performing the first operation with respect to thefirst user interface object (e.g., user interface object 7016, oranother user interface object, etc.) in accordance with the first userinput includes: in accordance with a determination that the first userinput includes (e.g., is, includes, starts with, ends with, etc.) apredefined selection gesture, selecting the first user interface objectas a target for a subsequent input (e.g., a drag gesture while the pinchgesture is maintained, a flick gesture while the pinch gesture ismaintained, a drag gesture after the predefined selection gesture isterminated, etc.) received from the first user (e.g., user 7102 in FIGS.7A-7C, or another user, etc.). In some embodiments, the selectiongesture is a pinch gesture that includes touch-down of an index fingeron a thumb of the same hand (optionally, followed by lifting off of theindex finger from the thumb, or flick of the wrist connected to thehand, or translation of the whole hand, etc.), a gesture that includesan index finger and a thumb of the same hand pulling apart from eachother from a touching posture, a pinch gesture, a pinch and draggesture, a pinch and flick gesture, etc. In some embodiments, if thefirst user's hand gets near a location that corresponds to a positionnear the first user interface object and performs the predefinedselection gesture, the computer system generates visual feedback toindicate that the first user interface object is not available forinteraction with the first user in accordance with a determination thatthe second user is currently interacting with the first user interfaceobject, and the computer system does not select the first user interfaceobject for subsequent interaction with the first user. If the computersystem determines that the second user is not currently interacting withthe first user interface object, the computer system does not displaythe visual feedback and selects the first user interface object forsubsequent interaction with the first user.

Selecting the first user interface object as a target for a subsequentinput received from the first user, in accordance with a determinationthat the first user input includes a predefined selection gesture,provides additional control options without cluttering the UI withadditional displayed controls (e.g., additional displayed controls forselecting the first user interface object). Providing additional controloptions without cluttering the UI with additional displayed controlsenhances the operability of the device, which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, in conjunction with selecting the first userinterface object (e.g., user interface object 7016 in FIGS. 7B-7C, oranother user interface object, etc.) as a target for a subsequent inputreceived from the first user (e.g., in response to detecting at least aportion of the first user input, in response to detecting the predefinedselection gesture provided by the first user while the second user isnot currently interacting with the first user interface object, etc.),the computer system displays a representation of the first userinterface object at a position that corresponds to a location of a handof the first user (e.g., the hand that performed the predefinedselection gesture or a hand of the first user that is different from thehand the performed the first predefined selection gesture, either handof the first user, etc.) (e.g., the representation or ghost image of thefirst user interface object is display near or at the position of therepresentation of the first user's hand, on the palm portion of therepresentation of the first user's hand, etc.), while maintaining thefirst user interface object at the first position in the first view ofthe three-dimensional environment (e.g., the first user interface objectremains at its original location, but can be “remotely” controlled bythe first user in accordance with interaction between the first user andthe representation of the first user interface object). In someembodiments, the representation of the first user interface object is aduplicate of the first user interface object, an image of the first userinterface object that is more translucent than the first user interfaceobject, a reduced version of the first user interface object that has asubset of the characteristics (e.g., a simplified internal structure, anoutline, etc.) and/or functions of the first user interface object(e.g., with reduced user interface elements, removal of textual content,etc.), etc. In some embodiments, when the first user interface object isselected in response to the selection gesture performed by the firstuser, a ghost image of the first user interface object is displayed at aposition that corresponds to the location of the hand of the first user(e.g., floating above the representation of the first user's hand, onthe representation of the user's palm, etc.).

Selecting the first user interface object as a target for a subsequentinput received from the first user in conjunction with displaying arepresentation of the first user interface object at a position thatcorresponds to a location of a hand of the first user, while maintainingthe first user interface object at the first position in the first viewof the three-dimensional environment, provides improved visual feedbackto the users (e.g., improved visual feedback that the first userinterface object is now a target for a subsequent input received fromthe first user). Providing improved feedback enhances the operability ofthe device, which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, after the first operation is performed with respectto the first user interface object in accordance with the first userinput (e.g., a representation of the first user interface object isdisplayed near the first position along with the first user interfaceobject, a representation of the first user interface object (e.g., userinterface object 7016 in FIGS. 7B-7C, or another user interface object,etc.) moves near a position of the representation of the first user'shand (e.g., representation 7202′ in the first view 7015-1 in FIGS.7B-7C, or another representation of the hand 7202, etc.), the first userinterface object moves toward the position of the first user, etc., toindicate that the first user interface object is ready for interactionwith the first user, etc.), the computer system detects a second userinput directed to the first user interface object (e.g., detecting apredefined pinch gesture directed to a position of the representation ofthe first user interface object, detecting a predefined pinch gesturedirected to a position of the first user interface object, detecting agaze input directed to the position of the representation of the firstuser interface object in conjunction with detecting a predefinedselection input, detecting a gaze input directed to a position of thefirst user interface object in conjunction with detecting a predefinedselection input, etc.). In response to detecting the second user input,in accordance with a determination that the second user input includes(e.g., is, includes, starts with, ends with, etc.) a predefinedselection gesture, the computer system selects the first user interfaceobject as a target for a subsequent input (e.g., a drag gesture whilethe pinch gesture is maintained, a flick gesture while the pinch gestureis maintained, a drag gesture after the predefined selection gesture isterminated, etc.) received from the first user.

In some embodiments, the selection gesture is a pinch gesture thatincludes touch-down of an index finger on a thumb of the same hand(optionally, followed by lifting off of the index finger from the thumb,or flick of the wrist connected to the hand, or translation of the wholehand, etc.), a gesture that includes an index finger and a thumb of thesame hand pulling apart from each other from a touching posture, a pinchgesture, a pinch and drag gesture, a pinch and flick gesture, etc. Insome embodiments, performing the first operation with respect to thefirst user interface object (e.g., user interface object 7106 in FIGS.7B-7C, or another user interface object, etc.) in accordance with thefirst user input includes displaying (e.g., at a second position that isdifferent from the first position at which the first user interfaceobject is displayed, and/or that is different from a position of arepresentation of the hand of the first user that provided the firstuser input, etc.) a representation of the first user interface objectwhile maintaining the first user interface object at the first positionin the first view of the three-dimensional environment. In someembodiments, the representation of the first user interface object is aduplicate of the first user interface object, an image of the first userinterface object that is more translucent than the first user interfaceobject, a reduced version of the first user interface object that has asubset of the characteristics (e.g., a simplified internal structure, anoutline, etc.) and/or functions of the first user interface object(e.g., with reduced user interface elements, removal of textual content,etc.), etc. In some embodiments, the representation of the first userinterface object is displayed slightly offset from the first userinterface object (e.g., floating above, in front of, etc. of the firstuser interface object in the first view of the three-dimensionalenvironment, not at a position that corresponds to the location of thefirst user's hand, etc.). In some embodiments, the representation of thefirst user interface object moves toward the representation of the firstuser's hand, but stays outside of the representation of the first user'shand.

Selecting the first user interface object as a target for a subsequentinput received from the first user, after the first operation isperformed and in accordance with a determination that the second userinput includes a predefined gesture, provides additional control optionswithout cluttering the UI with additional displayed controls (e.g.,additional displayed controls for selecting the first user interfaceobject). Providing additional control options without cluttering the UIwith additional displayed controls enhances the operability of thedevice, which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, performing the first operation with respect to thefirst user interface object (e.g., user interface object 7106 in FIGS.7B-7C, or another user interface object) in accordance with the firstuser input includes displaying (e.g., at a second position that isdifferent from the first position at which the first user interfaceobject is displayed, and/or that is different from a position of arepresentation of the hand of the first user that provided the firstuser input) a representation of the first user interface object (e.g., aduplicate of the first user interface object, an image of the first userinterface object that is more translucent than the first user interfaceobject, a reduced version of the first user interface object that has asubset of the characteristics (e.g., a simplified internal structure, oran outline) and/or functions of the first user interface object (e.g.,with reduced user interface elements, or reduced textual content)) whilemaintaining the first user interface object at the first position in thefirst view of the three-dimensional environment. In some embodiments,the representation of the first user interface object is displayedslightly offset from the first user interface object (e.g., floatingabove, in front of, etc. of the first user interface object in the firstview of the three-dimensional environment, not at a position thatcorresponds to the location of the first user's hand, etc.). In someembodiments, the representation of the first user interface object movestoward the representation of the first user's hand, but stays outside ofthe representation of the first user's hand. Displaying a representationof the first user interface object while maintaining the first userinterface object at the first position in the first view of thethree-dimensional environment provides improved visual feedback to theusers (e.g., improved visual feedback that the computer system isperforming the first operation with respect to the first user interfaceobject in accordance with the first user input). Providing improvedfeedback enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, the representation of the first user interfaceobject (e.g., user interface object 7106 in FIGS. 7B-7C, or another userinterface object, etc.) is initially displayed at a position that isaway from a representation of a hand of the first user (e.g., the hand7202 in FIGS. 7B-7C, or another hand, etc.) (e.g., the hand thatprovided the first user input, another hand of the first user, etc.),and the computer system moves the representation of the first userinterface object from the position that is away from the representationof the hand of the first user (e.g., the hand that provided the firstuser input, the other hand of the first user, etc.) to a position of therepresentation of the hand of the first user (e.g., showing therepresentation of the first user interface object flying from itsinitial display position to a position of the representation of thefirst user's hand, onto the representation of the user's palm, etc.) inaccordance with a determination that the first user interface object isselected by a subsequent user input provided by the first user. In someembodiments, the computer system moves the representation of the firstuser interface object in response to detecting a second user inputdirected to the first user interface object (e.g., detecting apredefined pinch gesture directed to the representation of the firstuser interface object, detecting a predefined pinch gesture directed tothe first user interface object, detecting a gaze input directed to therepresentation of the first user interface object in conjunction withdetecting a predefined selection input, or detecting a gaze inputdirected to the first user interface object in conjunction withdetecting a predefined selection input) that meets selection criteria,or that includes a selection gesture. Moving the representation of thefirst user interface object from the position that is away from therepresentation of the hand of the first user to a position of therepresentation of the hand of the first user in accordance with adetermination that the first user interface object is selected by asubsequent user input provided by the first user, provides improvedvisual feedback to the first user (e.g., improved visual feedback thatthe first user interface object is selected by the subsequent user inputprovided by the first user). Providing improved feedback enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, while displaying the representation of the firstuser interface object (e.g., the representation of the first userinterface object was displayed in response to detecting the first userinput (e.g., detecting the first user's hand moving near a location thatcorresponds to the position of the first user interface object in thethree-dimensional environment)), the computer system detects movement ofa hand of the first user (e.g., the hand 7202, or another hand of theuser 7102, etc.) (e.g., the hand that provided the first user input, ahand of the first user that is different from the hand that provided thefirst user input, etc.). In response to detecting the movement of thehand of the first user (e.g., the hand that provided the first userinput, a hand of the first user that is different from the hand thatprovided the first user input, etc.) and in accordance with adetermination that the movement of the hand of the first user meetspreset criteria for identifying an initial portion of a preset selectiongesture, the computer system changes an appearance of the representationof the first user interface object (e.g., the user interface object 7016in FIGS. 7B-7C, or another user interface object, etc.) (e.g., changingthe shape, size, color, opacity, level of details, etc., optionally, tomake the representation of the first user interface object more closelyresemble the appearance of the first user interface object, etc.). Insome embodiments, the appearance of the representation of the first userinterface object continues to change when the movement of the firstuser's hand continue to conform to the required progress of theselection gesture. In some embodiments, in response to detecting thatthe movement of the hand of the first user does not meet the presetcriteria for identifying the initial portion of the preset selectiongesture, the computer system does not change the appearance of therepresentation of the first user interface object or change theappearance in a different manner to indicate to the first user that thepreset selection gesture is not being detected. In some embodiments, inresponse to detecting that the movement of the first user's hand is nolonger conforming to the required progress of the preset selectiongesture, the computer system ceases to change the appearance of therepresentation of the first user interface object, and restores theappearance of the first user interface object or ceases to display therepresentation of the first user interface object.

Changing an appearance of the representation of the first user interfaceobject in accordance with a determination that the movement of the handof the first user meets preset criteria for identifying an initialportion of a preset selection gesture, provides improved visual feedbackto the first user (e.g., improved visual feedback that the computersystem has detected movement of the hand of the first user that meetsthe preset criteria). Providing improved feedback enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, while displaying the representation of the firstuser interface object (e.g., user interface object 7106 in FIGS. 7B-7C,or another user interface object, etc.) at the position of therepresentation of the hand of the first user (e.g., hand 7202 in FIGS.7B-7C, or another hand, etc.) (e.g., after showing the representation ofthe first user interface object flying from its initial display positionto the position of the representation of the first user's hand, onto therepresentation of the user's palm, etc.) and in accordance with adetermination that the first user interface object is selected by thefirst user, the computer system detects a third user input interactingwith the representation of the first user interface object (e.g., thirdinput is an input provided by the hand that provided the first userinput, the hand that selected the representation of the first userinterface object, a hand of the first user that is different from thehand that provided the first user input, or a hand of the first userthat is different from the hand that selected the representation of thefirst user interface object, etc.). In response to detecting the thirduser input, the computer system displays visual feedback for the thirduser input through at least one of movement of the representation of thefirst user interface object and changing appearance of therepresentation of the first user interface object, and the computersystem performs a second operation with respect to the first userinterface object in accordance with the third user input. In someembodiments, the visual feedback corresponds to a second operation thatis to be performed with respect to the first user interface object. Insome embodiments, the visual feedback corresponds to direct manipulationof the representation of the first user interface object in accordancewith the movement of the hand of the first user that is at a locationthat corresponds to the position of the representation of the first userinterface object. In some embodiments, performing the second operationincludes activating a control on the first user interface object, movingthe first user interface object in three-dimensional space, playing backof media item represented by the first user interface object, launchingan application represented by the first user interface object, and/orstarting a communication session with another user represented by thefirst user interface object, etc.

Displaying visual feedback for the third user input through at least oneof movement of the representation of the first user interface object andchanging appearance of the representation of the first user interfaceobject, and performing a second operation with respect to the first userinterface object in accordance with the third user input, in response todetecting the third user input interacting with the representation ofthe first user interface object, provides improved visual feedback tothe users (e.g., improved visual feedback that the computer system hasdetected the third user input, and/or improved visual feedback that thecomputer system is performing the second operation with respect to thefirst user interface object). Providing improved feedback enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the computer system updates the position of therepresentation of the first user interface object (e.g., user interfaceobject 7106 in FIGS. 7B-7C, or another user interface object, etc.) inaccordance with movement of the hand of the first user such that therepresentation of the first user interface object maintains an existingspatial relationship with (e.g., stays fixed to, follows, etc.) theupdated position of the representation of the hand of the first user(e.g., hand 7202 in FIGS. 7B-7C, or another hand, etc.) (e.g., while thefirst user interface remains at the first position in the first view ofthe three-dimensional environment). In some embodiments, therepresentation of the first user interface object remains displayed at aposition at or near the representation of the hand of the first user,irrespective of whether the first user actually interacts with therepresentation of the first user interface object. In some embodiments,the representation of the first user interface object is displayed at aposition that corresponds to the location of the palm of the hand of thefirst user, and ceases to be displayed when the representation of thehand is not in the currently displayed view of the three-dimensionalenvironment, or when the hand of the first user is closed. In someembodiments, the representation of the first user interface object isredisplayed when the representation of the hand of the first user, orwhen the representation of the palm of the hand is visible again in thecurrently displayed view of the three-dimensional environment. The firstuser can interact with the representation of the first user interfaceobject as long as the first user has not specifically relinquishedcontrol of the first user interface object (e.g., by providing athrowing gesture toward the second user, or toward a shared portion ofthe three-dimensional environment (e.g., a representation of a wall or atable top, etc.), etc.). In some embodiments, the first userrelinquishes control of the first user interface object when the firstuser closes his/her hand so that the representation of the first userinterface object is no longer displayed in the currently displayed viewof the three-dimensional environment (e.g., the first user has to regaincontrol by starting over again with another selection input while thesecond user is not interacting with the first user interface object).

Updating the position of the representation of the first user interfaceobject in accordance with movement of the hand of the first user suchthat the representation of the first user interface object maintains anexisting spatial relationship with the updated position of therepresentation of the hand of the first user provides improved visualfeedback to the users (e.g., improved visual feedback that the firstuser interface object is selected by a subsequent user input provided bythe first user). Providing improved feedback enhances the operability ofthe device, which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

It should be understood that the particular order in which theoperations in FIG. 8 have been described is merely an example and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 9000, 10000, 11000, and 12000) are also applicable in ananalogous manner to method 8000 described above with respect to FIG. 8 .For example, the gestures, gaze inputs, physical objects, user interfaceobjects, controls, movements, criteria, three-dimensional environment,display generation component, surface, representation of physicalobject, virtual objects, and/or animations described above withreference to method 8000 optionally have one or more of thecharacteristics of the gestures, gaze inputs, physical objects, userinterface objects, controls, movements, criteria, three-dimensionalenvironment, display generation component, surface, representation ofphysical object, virtual objects, and/or animations described hereinwith reference to other methods described herein (e.g., methods 9000,10000, 11000, and 12000). For brevity, these details are not repeatedhere.

FIGS. 9A-9B are a flowchart of a method of displaying a representationof a physical object relative to a viewpoint of a currently displayedview of a three-dimensional environment in different manners, where theviewpoint moves in accordance with movement of the user in a firstphysical environment, the representation of the physical object moves inaccordance with movement of the physical object in a second physicalenvironment different from the first physical environment, and where achange in the manner of displaying the representation is triggered inresponse to a spatial relationship between the representation of thephysical object and the viewpoint meeting preset criteria, in accordancewith some embodiments.

In some embodiments, the method 9000 is performed at a computer system(e.g., computer system 101 in FIG. 1 ) including a display generationcomponent (e.g., display generation component 120 in FIGS. 1, 3, and 4 )(e.g., a heads-up display, a display, a touchscreen, a projector, etc.)and one or more cameras (e.g., a camera (e.g., color sensors, infraredsensors, and other depth-sensing cameras) that points downward at auser's hand or a camera that points forward from the user's head). Insome embodiments, the method 9000 is governed by instructions that arestored in a non-transitory computer-readable storage medium and that areexecuted by one or more processors of a computer system, such as the oneor more processors 202 of computer system 101 (e.g., control unit 110 inFIG. 1A). Some operations in method 9000 are, optionally, combinedand/or the order of some operations is, optionally, changed.

In some embodiments, the method 9000 is performed at a computer system(e.g., computer system 101 in FIG. 1 ) that is in communication with adisplay generation component (e.g., display generation component 120 inFIGS. 1, 3, and 4 , display generation component 7100, etc.) (e.g., aheads-up display, an HMD, a display, a touchscreen, a projector, etc.)and one or more input devices (e.g., cameras, controllers,touch-sensitive surfaces, joysticks, buttons, etc.). In someembodiments, the computer system is an integrated device with one ormore processors and memory enclosed in the same housing as the displaygeneration component and at least some of the one or more input devices.In some embodiments, the computer system includes a computing componentthat includes one or more processors and memory that is separate fromthe display generation component and/or the one or more input devices.In some embodiments, the display generation component and the one ormore input devices are integrated and enclosed in the same housing.

In the method 9000, while a first user (e.g., user 7002 in FIGS. 7D-7F,or another user) is at a first location in a first physical environment(e.g., scene 105-a in FIGS. 7D-7F, or another physical environment,etc.), the computer system displays (9002) a first view of athree-dimensional environment (e.g., a view 7304-a in FIG. 7D) (e.g., avirtual three-dimensional environment, an augmented reality environment,a three-dimensional mixed reality environment, etc.) corresponding to afirst viewpoint that is associated with the first location in the firstphysical environment (e.g., the first viewpoint is a virtual positionthat corresponds to the location of the user's eyes or face when thefirst user is at the first location in the physical environment, thefirst viewpoint corresponds to a first viewing perspective toward thethree-dimensional environment, etc.), wherein the first view of thethree-dimensional environment includes a first user interface object(e.g., representation 7102′-a of the second user 7102 in FIGS. 7D-7F, oranother representation of another physical object, etc.) (e.g., anavatar for a second user that is different from the first user, avirtual representation of a moving physical object (e.g., an animal, avehicle, a flying drone, an opponent in a multiplayer game, etc.)) thatrepresents a first object (e.g., a second user, a moving physical object(e.g., an animal, a flying drone, an opponent in a game, etc.), etc.) ina second physical environment (e.g., scene 105-b in FIGS. 7D-7F, oranother physical environment, etc.) different from the first physicalenvironment (e.g., the first physical environment and the secondphysical environment are in two different rooms, at two differentgeographical locations, etc., where the first user and the second userare out of physical reach of each other (e.g., no risk of physicalcollision) when respectively located in the first physical environmentand the second physical environment), wherein a respective position ofthe first user interface object in the three-dimensional environmentcorresponds to a respective location of the first object in the secondphysical environment in a first manner (e.g., movement distance andmovement direction of the first object (e.g., the second user, ananimal, a flying drone, etc.) in the second physical environment arerespectively mapped to movement distance and movement direction of thefirst user interface object in the three-dimensional environment inaccordance with a first preset object-mapping relationship (e.g., alinear mapping relationship, and/or in the same type of coordinatesystem with the same cardinal directions, etc.), movement distance andmovement direction of the respective position corresponding to theviewpoint of the currently displayed view of the three-dimensionalenvironment (e.g., the viewpoint of the first user, the virtual positionof the first user in the three-dimensional environment, etc.) arerespectively mapped to movement distance and movement direction of thefirst user in the first physical environment in accordance with a firstpreset user-mapping relationship (e.g., a linear mapping relationship,and/or in the same type of coordinate system with the same cardinaldirections, etc.), etc.). In some embodiments, the correspondence in thefirst manner is the default correspondence (e.g., reality-mimickingcorrespondence, a correspondence that also applies to movement of onepart of the first user to another part of first user in front of thefirst user, etc.) between the motion and location in the real-world andthe motion and locations in the three-dimensional environment. In someembodiments, the three-dimensional environment is a virtual environmentor augmented reality environment that is at least partially sharedbetween the first user and the second user, where the first user and thesecond user may view the same portion of the three-dimensionalenvironment from two different viewpoints relative to thethree-dimensional environment.

In some embodiments, the three-dimensional environment is not a sharedenvironment. In some embodiments, the three-dimensional environment is avirtual three-dimensional environment. In some embodiments, thethree-dimensional environment is an augmented reality environment thatincludes a representation of the first physical environment (e.g., acamera view or transparent pass-through view of the first physicalenvironment, etc.) and optionally a representation of a physical object(e.g., the second user, an animal, a flying drone, etc.) in the secondphysical environment (e.g., without including a representation of thesecond physical environment). In some embodiments, the three-dimensionalenvironment is an augmented reality view that includes a representationof the second physical environment (e.g., a camera view or transparentpass-through view of the second physical environment, a video recordingcaptured at the second physical environment, etc.) with a camera view orrecorded image of the first object removed and replaced by the firstuser interface object (e.g., so that the position of the first userinterface object can be modified computationally or digitally relativeto the representation of the second physical environment in theaugmented reality environment shown via the first display generationcomponent). In some embodiments, the movement of the first user as awhole in the first physical environment (e.g., walking, running, ridinga bike, jumping upward, riding an elevator, etc. in the first physicalenvironment, instead of merely moving the first user's head or armswithout moving the whole person in the first physical environment)causes a corresponding change in the viewpoint of the currentlydisplayed view of the three-dimensional environment (e.g., translationof the viewpoint relative to the three-dimensional environment in acorresponding direction and/or with a corresponding distance, etc.); andmovement of the first object (e.g., the second user as a whole, ananimal, a flying drone, etc.) in the second physical environment (e.g.,walking, running, riding a bike, jumping upward, riding an elevator,flying, etc.) causes movement of the first user interface object in acorresponding direction and/or with a corresponding distance, etc. inthe three-dimensional environment.

In some embodiments, when the movement and/or position of the first userinterface object is determined based on the movement and/or location ofthe first object in the second physical environment in the first manner(e.g., without regard to the current location and/or movement history ofthe first user in the first physical environment), the first userinterface object may end up at a position in the three-dimensionalenvironment that will be within a threshold distance of the viewpoint ofthe currently displayed view of the three-dimensional environment shownvia the first display generation component (e.g., the viewpointdetermined in accordance with the current location and movement historyof the first user in the first physical environment). In someembodiments, having the first user interface object displayed within thethreshold distance of the virtual position of the viewpoint of thecurrently displayed view of the three-dimensional environment wouldresult in the first user interface object appearing very large,unnatural, and/or intrusive to the personal space of the viewer of thethree-dimensional environment (e.g., the first user). In someembodiments, the currently displayed view of the three-dimensionalenvironment shown via the first display generation component does notvisibly include a virtual representation of the first user's body (e.g.,the virtual position of the first user relative to the three-dimensionalenvironment is reflected by the currently displayed view itself and thecorresponding viewpoint). In some embodiments, the currently displayedview of the three-dimensional environment shown via the first displaygeneration component visibly includes a virtual representation of aportion of the first user's body (e.g., the view includes at the bottomof the view a representation of the first user's outline, the firstuser's hands or feet in front of the first user's eyes, the viewincludes the first user's avatar whose position in the three-dimensionalenvironment is determined based on the movement and/or current locationof the first user in the first physical environment in the first mannerand stays stationary relative to the display (e.g., has a fixed spatialrelationship with the viewpoint of the currently displayed view of thethree-dimensional environment), etc.).

In the method 9000, the computer system detects (9004) at least one of(e.g., only one of, only of the first user, only of the first object(e.g., the second user, an animal, a flying drone, etc.), or both of,etc.) movement of the first user (e.g., first user 7002 in FIGS. 7D-7F)in the first physical environment (e.g., movement from a first locationto a second location, in a first direction (e.g., forward, backward,leftward, rightward, upward, downward, in the 2 o'clock direction, inthe 10 o'clock direction, etc.), by a first distance, and/or with afirst speed, etc. in the first physical environment) and movement of thefirst object (e.g., the second user 7102 in FIGS. 7D-7F, or anotherphysical object, etc.) in the second physical environment (e.g.,movement from a third location to a fourth location, in a seconddirection (e.g., forward, background, leftward, rightward, upward,downward, in the 3 o'clock direction, in the 8 o'clock direction, etc.),by a second distance, and/or with a second speed, etc.). In someembodiments, detecting the movement of the first user is performeddirectly using sensors collocated with the first user and the firstdisplay generation component, and, optionally, detecting the movement ofthe first object in the second physical environment is performedindirectly (e.g., through the sensors collocated with the first objectin the second physical environment, through receipt of an updatedposition of the first user interface object transmitted from thecomputer system used at the second physical environment (e.g., used bythe second user, or another user different from the first and secondusers, etc.), where the computer system has updated the position of thefirst user interface object in accordance with the first manner (e.g.,according to the preset first reality-mimicking mapping relationship),etc.).

In the method 9000, in response to detecting (9006) the at least one ofmovement of the first user in the first physical environment andmovement of the first object in the second physical environment (e.g.,in response to detecting only the movement of the first user, inresponse to detecting only the movement of the first object, in responseto detecting either one of the movement of the user or the object, inresponse to detecting both the movement of the user and the movement ofthe object, etc.): the computer system displays (9008) a second view ofthe three-dimensional environment corresponding to a second viewpoint(e.g., view 7304-a′ in FIG. 7E, view 7304-a″ in FIG. 7F, etc.) (e.g.,the second viewpoint is the same as the first viewpoint if the firstuser has not moved in the first physical environment, and the secondviewpoint is different from the first viewpoint if the first user hasmoved in the first physical environment (e.g., the movement of the firstuser in the physical environment is mapped to movement of the first userwithin the three-dimensional environment, which causes shifting of theviewpoint of the view of the three-dimensional environment first userthat is displayed to the first user), etc.); and the computer systemdisplays (9010) the first user interface object (e.g., representation7102′-a in FIGS. 7E and 7F) in the second view of the three-dimensionalenvironment (e.g., view 7304-a′ in FIG. 7E, view 7304-a″ in FIG. 7F,etc.). Displaying the first user interface object includes: inaccordance with a determination that the respective position of thefirst user interface object in the three-dimensional environment thatcorresponds to the respective location of the first object in the secondphysical environment in the first manner (e.g., determined in accordancewith a reality-mimicking manner, with the preset first mappingrelationship, etc.) (e.g., the same previous position of the first userinterface object in the three-dimensional environment if the first userinterface object has been substantially stationary in thethree-dimensional environment; a different position in thethree-dimensional environment if the first user interface object hasbeen moving in the three-dimensional environment, etc.) is more than athreshold distance (e.g., more than an arm's length, more than a presetradius of a personal space for the first user in the three-dimensionalenvironment, outside of a preset boundary surface surrounding a virtualposition of the first user in the three-dimensional environment (e.g.,the virtual surface of the representation of the first user, a boundingbox surrounding the virtual position of the first user), etc.) from arespective position in the three-dimensional environment thatcorresponds to the second viewpoint associated with the second view ofthe three-dimensional environment (e.g., the respective position is thevirtual position of the first user in the three-dimensional environment(e.g., a visible position, or an invisible position in the currentlydisplayed view of the three-dimensional environment), the respectiveposition is the virtual position of the head of the first user orhis/her virtual representation in the three-dimensional environment, therespective position is one of one or more positions on a boundarysurface surrounding a virtual position of the first user or his/hervirtual representation in the three-dimensional environment, etc.),displaying (9012) the first user interface object at a first displayposition in the second view of in the three-dimensional environment,wherein the first display position is the respective position of thefirst user interface object in the three-dimensional environment. Thisis a scenario illustrated in FIG. 7E, for example, where the displayposition of the representation 7102′-a of the second user is the same asthe respective position of the representation 7102′-a calculated inaccordance with the first manner. For example, in some embodiments, thefirst display position is determined based on the current locationand/or movement history of the first object (e.g., the second user, ananimal, a flying drone, etc.) in the second physical environment in thefirst manner (e.g., independent of the current location and/or movementhistory of the first user in the first physical environment, inaccordance with the first preset mapping relationship, etc.); the firstdisplay position is the position of the first user interface object whenthe first user interface object has not gotten too close to theviewpoint of the currently displayed view of the three-dimensionalenvironment (or has not gotten too close to the virtual position offirst user and/or too close to the position of the virtualrepresentation of the first user in the three-dimensional environment,etc.) so as to appear too large, unnatural, and/or invasive to thepersonal space of the first user, etc.

Displaying the first user interface object further includes: inaccordance with a determination that the respective position of thefirst user interface object in the three-dimensional environment thatcorresponds to the respective location of the first object in the secondphysical environment in the first manner is less than the thresholddistance from the respective position in the three-dimensionalenvironment that corresponds to the second viewpoint associated with thesecond view of the three-dimensional environment, displaying (9014) thefirst user interface object at a second display position in the secondview of the three-dimensional environment, wherein the second displayposition is offset from the respective position (e.g., shifted sideways,shifted by more than a threshold distance in a respective direction(e.g., sideways, upward, downward, etc.), shifted by more than thethreshold distance in a sideway direction (e.g., left, right, etc.)relative to the currently displayed view of the three-dimensionalenvironment, shifted by more than the threshold distance in an upwarddirection relative to the currently displayed view of thethree-dimensional environment, etc.) of the first user interface objectin the three-dimensional environment (e.g., the second display positionis determined based on not only the current location and/or movementhistory of the first object (e.g., the second user, an animal, a flyingdrone, etc.) in the second physical environment, but also the currentlocation and/or movement history of the first user in the first physicalenvironment; the second display position is determined in accordancewith a second manner different from the first manner; the second displayposition is shifted from the default position of the first userinterface object when the first user interface object has gotten tooclose to the viewpoint of the currently displayed view of thethree-dimensional environment, so that the first user interface objectdoes not appear too large, unnatural, and/or invasive to the personalspace of the first user, etc.). This is a scenario illustrated in FIG.7F, for example, where the display position of the representation7102′-a of the second user is offset from the respective position of therepresentation 7102′-a calculated in accordance with the first mannerand is calculated in accordance with a second manner.

In some embodiments, to determine the position of the first userinterface object in accordance with the second manner different from thefirst manner, the movement distance and movement direction of the firstobject (e.g., the second user, an animal, a flying drone, etc.) in thesecond physical environment are respectively mapped to movement distanceand movement direction of the first user interface object in thethree-dimensional environment in accordance with a second presetobject-mapping relationship (e.g., a non-linear mapping relationship, alinear-mapping relationship with an additional linear or non-linearoffset amount that is based on the current position, the movementdistance, and/or movement direction of the respective positioncorresponding to the viewpoint of the currently displayed view of thethree-dimensional environment (e.g., the viewpoint of the first user,the virtual position of the first user in the three-dimensionalenvironment, etc.) and/or the current position, the movement distanceand/or movement direction of the first user interface object in thethree-dimensional environment. In some embodiments, the correspondencein the second manner includes a modification to the defaultcorrespondence (e.g., reality-mimicking correspondence) between themotion and location in the real-world and the motion and locations inthe three-dimensional environment, with the purpose to avoid having thefirst user interface object appear too close in the first user's view ofthe three-dimensional environment (e.g., a visual-collision-avoidancecorrespondence). In some embodiments, the direction and amount by whichthe second display position of the first user interface object isshifted or offset from the default display position (e.g., the positiondetermined in accordance with the reality mimicking correspondence, inaccordance with the first preset mapping relationship, etc.) isdetermined in accordance with the size and/or shape of the first objectand/or the size and/or shape of the first user, a size and/or shape of abounding box associated with the first user, a size and/or shapeassociated with a bounding box of the first object, and/or a size and/orshape of a virtual representation of the first user in thethree-dimensional environment, etc. In a first example, in someembodiments, the first object is a second user, and when the first userand/or the second user walk within their respective physicalenvironments such that the respective position of the virtualrepresentation of the second user in the currently displayed view of thethree-dimensional environment (as calculated in accordance with thefirst, reality-mimicking mapping relationship) is beyond the thresholddistance of the viewpoint corresponding to the currently displayed view,the virtual representation of the second user is displayed at therespective position calculated in accordance with the first,reality-mimicking mapping relationship in the currently displayed viewof the three-dimensional environment (e.g., based on the currentlocation and movement history of the second user in the second physicalenvironment without consideration of the current location and movementhistory of the first user in the first physical environment). However,when the first user and/or the second user walk within their respectivephysical environment such that the respective position of the virtualrepresentation of the second user in the currently displayed view of thethree-dimensional environment as calculated in accordance with thefirst, reality-mimicking mapping relationship would fall within thethreshold distance of the viewpoint corresponding to the currentlydisplayed view, the displayed position of the virtual representation ofthe second user is shifted from the respective position (e.g., aposition that is calculated based on the current location and movementhistory of the second user in the second physical environment withoutconsideration of the current location and movement history of the firstuser in the first physical environment) such that the virtualrepresentation of the second user would not appear to bump into thefirst user and/or overwhelm the view of the first user from the viewingperspective of the first user, and/or would not overlap with the virtualrepresentation of the first user in the three-dimensional environment(e.g., visible virtual representation, or a virtual representation thatis not visible in the currently displayed view of the three-dimensionalenvironment, etc.). In some embodiments, even though the displayedposition of the representation of the second user is shifted in the viewof the three-dimensional environment, the respective position of therepresentation of the second user in the three-dimensional environmentis not shifted (it is just not visually reflected in the displayed viewof the three-dimensional environment shown to the first user). The abovefeatures are illustrated, for example, in FIGS. 7D-7F, in accordancewith some embodiments. In FIG. 7E, for example, in the view 7304-a′shown to the first user 7002, the display position of the representation7102′-a of the second user 7102 is the same as the respective positionof the representation 7102′-a calculated in accordance with a first typeof correspondence between positions in the three-dimensional environmentand the locations in the second physical environment of the second user7102. This is the usual scenario, where the representation 7102′-a ofthe second user 7102 is not within a threshold distance of the viewpointof the first view 7304-a shown to the first user 7002 (the left view inFIG. 7E). In FIG. 7F, for example, in the view 7034-a″ shown to thefirst user 7002, the display position of the representation 7102′-a isoffset from the respective position of the representation 7102′-acalculated in accordance with the first type of correspondence, becausethe respective position would place the representation 7102′-a of thesecond user too close to the viewpoint of the first view 7304-a″ shownto the first user 7002. Therefore, in the scenario shown in FIG. 7F, therepresentation 7102′-a is shown to be on the right side of therepresentation 7002′-a of the first user 7002, even though therespective position of the representation 7102′-a of the second user isin fact straight in front of the representation 7002′-a in the firstview 7304-a″.

In some embodiments, detecting the at least one of movement of the firstuser (e.g., user 7002 in FIGS. 7D-7F, or another user, etc.) in thefirst physical environment (e.g., scene 105-a in FIGS. 7D-7F, or anotherscene, etc.) and movement of the first object (e.g., user 7102 in FIGS.7D-7F, or another person or object, etc.) in the second physicalenvironment (e.g., scene 105-b in FIGS. 7D-7F, or another scene, etc.)includes detecting first movement of the first user (e.g., user 7002 inFIGS. 7D-7F, or another user, etc.) in the first physical environmentwhile the first object (e.g., user 7102 in FIGS. 7D-7F, or anotherperson or object, etc.) remains stationary in the second physicalenvironment. During the first movement of the first user in the firstphysical environment (e.g., during movement of the first user as a wholethat causes movement of the viewpoint toward a representation of astationary second user or virtual object in the three-dimensionalenvironment): in accordance with the determination that the respectiveposition of the first user interface object in the three-dimensionalenvironment that corresponds to the respective location of the firstobject in the second physical environment in the first manner (e.g., astationary position in the three-dimensional environment thatcorresponds to the stationary location of the first object in the secondphysical environment during the first movement of the first user) ismore than the threshold distance from the respective position in thethree-dimensional environment that corresponds to a viewpoint associatedwith a currently displayed view of the three-dimensional environment(e.g., the representation of the second user or virtual object isoutside of a threshold range of the first user's viewpoint or virtualposition in the three-dimensional environment), the first user interfaceobject is displayed at the respective position of the first userinterface object in the three-dimensional environment that correspondsto the respective location of the first object in the second physicalenvironment in the first manner (e.g., the appearance and location ofthe first user interface object may vary visually to the first user(e.g., appearing with different sizes or in different portions of thefield of view of the first user,) due to the changing viewpoint, whilethe respective three-dimensional position of the first user interfaceobject in the three-dimensional environment does not change). During thefirst movement of the first user in the first physical environment(e.g., during movement of the first user as a whole that causes movementof the viewpoint toward a representation of a stationary second user orvirtual object in the three-dimensional environment): in accordance withthe determination that the respective position of the first userinterface object in the three-dimensional environment that correspondsto the respective location of the first object in the second physicalenvironment in the first manner (e.g., a stationary position in thethree-dimensional environment that corresponds to the stationarylocation of the first object in the second physical environment duringthe first movement of the first user) is not more than (e.g., equal to,less than, etc.) the threshold distance from the respective position inthe three-dimensional environment that corresponds to the viewpointassociated with the currently displayed view of the three-dimensionalenvironment (e.g., the representation of the second user or virtualobject will be within the threshold range of the first user's viewpointor virtual position in the three-dimensional environment, if nototherwise shifted away), the first user interface object is displayed atan adjusted position (e.g., a position that is offset from therespective position that is determined based on the reality-mimickingmapping relationship, a position that is outside of the thresholddistance from the virtual representation of the first user or theviewpoint of the currently displayed view of the three-dimensionalenvironment, etc.) in the three-dimensional environment while the firstobject remains stationary in the second physical environment, whereinthe adjusted position in the three-dimensional environment correspondsto the respective location of the first object in the second physicalenvironment in a second manner different from the first manner. Forexample, in some embodiments, even though the first object remainsstationary in the second physical environment, the first user interfaceobject appears to make an autonomous movement in the currently displayedview of the three-dimensional environment in response to the movement ofthe first user (e.g., a movement in the first physical environment thatcorresponds to a movement of the viewpoint or virtual representation ofthe first user toward the respective position of the first userinterface object in the three-dimensional environment). In a morespecific example, when the first user makes a movement in the firstphysical environment that corresponds to a movement of the virtualrepresentation of the first user toward a representation of a seconduser or a virtual object in the three-dimensional environment, if thevirtual representation of the first user gets to within a thresholddistance of the representation of the second user or the virtual object,the representation of the second user or the virtual objectautomatically moves out of the way from the virtual representation ofthe first user to avoid bumping into the virtual representation of thefirst user, even if the second user or virtual object remains stationaryin the second physical environment. The representation of the seconduser and virtual object is restored to their default position calculatedusing the default reality-mimicking mapping relationship after thecondition for triggering the collision avoidance offset is no longermet. In some embodiments, the first user interface object is an avatarrepresenting a second user that is in a virtual conference call with thefirst user; when the first user walks in a direction that corresponds tomovement in the three-dimensional environment toward the avatar of thesecond user, the avatar of the second user moves out of the way as theavatar gets too close to the virtual position of the first user. This isdone even if the second user has not moved in his/her own environment.

Detecting first movement of the first user in the first physicalenvironment while the first object remains stationary in the secondphysical environment, and displaying the first user interface object atthe respective position of the first user interface object in thethree-dimensional environment that corresponds to the respectivelocation of the first object in the second physical environment in thefirst manner in accordance with the determination that the respectiveposition of the first user interface object in the three-dimensionalenvironment that corresponds to the respective location of the firstobject in the second physical environment in the first manner is morethan the threshold distance from the respective position in thethree-dimensional environment that corresponds to a viewpoint associatedwith a currently displayed view of the three-dimensional environment,and displaying the first user interface object at an adjusted positionin the three-dimensional environment corresponding to the respectivelocation of the first object in the second physical environment in asecond manner different from the first manner, while the first objectremains stationary in the second physical environment, in accordancewith the determination that the respective position of the first userinterface object in the three-dimensional environment that correspondsto the respective location of the first object in the second physicalenvironment in the first manner is not more than the threshold distancefrom the respective position in the three-dimensional environment thatcorresponds to the viewpoint associated with the currently displayedview of the three-dimensional environment, displays the first userinterface object at an appropriate position when a set of conditions hasbeen met without requiring further user input (e.g., further user inputto adjust the position of the first user interface object). Performingan operation when a set of conditions has been met without requiringfurther user input enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, detecting the at least one of movement of the firstuser (e.g., user 7002 in FIGS. 7D-7F, or another user, etc.) in thefirst physical environment (e.g., scene 105-a in FIGS. 7D-7F, or anotherscene, etc.) and movement of the first object (e.g., user 7102 in FIGS.7D-7F, or another person or object, etc.) in the second physicalenvironment (e.g., scene 105-b, or another scene, etc.) includesdetecting second movement of the first object in the second physicalenvironment while the first user remains stationary in the firstphysical environment. During the second movement of the first object inthe second physical environment (e.g., during movement of the firstobject (e.g., a second user, an animal, a flying drone, etc.) as a wholethat causes movement of the first user interface object in thethree-dimensional environment): in accordance with the determinationthat the respective position of the first user interface object in thethree-dimensional environment that corresponds to the respectivelocation of the first object in the second physical environment in thefirst manner (e.g., a new position in the three-dimensional environmentthat corresponds to the new location of the first object in the secondphysical environment during the second movement of the first object) ismore than the threshold distance from the respective position in thethree-dimensional environment that corresponds to a viewpoint associatedwith a currently displayed view of the three-dimensional environment(e.g., the representation of the second user or virtual object isoutside of a threshold range of the first user's viewpoint or virtualposition in the three-dimensional environment), the first user interfaceobject is displayed at (e.g., the first user interface object is shownto move to) an updated position (e.g., a position that is the respectiveposition that is determined based on the reality-mimicking mappingrelationship, a position that is outside of the threshold distance fromthe virtual representation of the first user or the viewpoint of thecurrently displayed view of the three-dimensional environment, etc.) inthe three-dimensional environment (e.g., while the viewpoint of thecurrently displayed view of the three-dimensional environment remainsstationary), wherein the updated position in the three-dimensionalenvironment corresponds to an updated location of the first object inthe second physical environment as a result of the second movement inthe first manner (e.g., the appearance and displayed position of thefirst user interface object change due to the change in the respectivethree-dimensional position of the first user interface object in thethree-dimensional environment, while the viewpoint of the currentlydisplayed view is maintained). During the second movement of the firstobject in the second physical environment (e.g., during movement of thefirst object (e.g., a second user, an animal, a flying drone, etc.) as awhole that causes movement of the first user interface object in thethree-dimensional environment): in accordance with the determinationthat the respective position of the first user interface object in thethree-dimensional environment that corresponds to the respectivelocation of the first object in the second physical environment in thefirst manner (e.g., a new position in the three-dimensional environmentthat corresponds to the new location of the first object in the secondphysical environment during the second movement of the first object) isnot more than (e.g., equal to, less than, etc.) the threshold distancefrom the respective position in the three-dimensional environment thatcorresponds to the viewpoint associated with the currently displayedview of the three-dimensional environment (e.g., the representation ofthe second user or virtual object will be within the threshold range ofthe first user's viewpoint or virtual position in the three-dimensionalenvironment, if not otherwise shifted away), the first user interfaceobject is displayed at (e.g., the first user interface object is shownto move to) an adjusted updated position (e.g., a position that isoffset from the respective position that is determined based on thereality-mimicking mapping relationship, a position that is outside ofthe threshold distance from the virtual representation of the first useror the viewpoint of the currently displayed view of thethree-dimensional environment, etc.) in the three-dimensionalenvironment while the first object remains stationary in the secondphysical environment, wherein the adjusted updated position in thethree-dimensional environment corresponds to the updated location of thefirst object in the second physical environment in a second mannerdifferent from the first manner. For example, in some embodiments, whenthe second user or virtual object moves in a direction in the secondphysical environment that corresponds to a movement toward the viewpointof the currently displayed view or the virtual position of the firstuser in the three-dimensional environment, the first user interfaceobject appears to be under the influence of a simulated physical forcesuch as magnetic or electrostatic repulsion that pushes it away fromand/or that prevents it from getting too close to the viewpoint or thevirtual position of the first user when the representation of the seconduser or virtual object is about to cross the threshold distance of theviewpoint or the virtual position of the first user in thethree-dimensional environment. The representation of the second user andvirtual object is restored to their original movement path calculatedusing the default reality-mimicking mapping relationship after thecondition for triggering the visual collision avoidance offset is nolonger met. In some embodiments, the first user interface object is afloating talking avatar representing a second user that is in a virtualconference call with the first user; when the second user walks in adirection that corresponds to movement in the three-dimensionalenvironment toward the virtual position of the first user, the floatingtalking avatar of the second user moves around the virtual position ofthe virtual position of the first user and does not ever get too closeto the virtual position of the first user. This is a modificationperformed on top of the usual movement calculated based on the defaultmapping relationship between movement of the second user in the secondphysical environment and movement of the second user's avatar in thethree-dimensional environment.

Detecting second movement of the first object in the second physicalenvironment while the first user remains stationary in the firstphysical environment, and displaying the first user interface object atthe respective position of the first user interface object in thethree-dimensional environment that corresponds to the respectivelocation of the first object in the second physical environment in thefirst manner in accordance with the determination that the respectiveposition of the first user interface object in the three-dimensionalenvironment that corresponds to the respective location of the firstobject in the second physical environment in the first manner is morethan the threshold distance from the respective position in thethree-dimensional environment that corresponds to a viewpoint associatedwith a currently displayed view of the three-dimensional environment,and displaying the first user interface object at an adjusted positionin the three-dimensional environment corresponding to the respectivelocation of the first object in the second physical environment in asecond manner different from the first manner, while the first userremains stationary in the second physical environment, in accordancewith the determination that the respective position of the first userinterface object in the three-dimensional environment that correspondsto the respective location of the first object in the second physicalenvironment in the first manner is not more than the threshold distancefrom the respective position in the three-dimensional environment thatcorresponds to the viewpoint associated with the currently displayedview of the three-dimensional environment, displays the first userinterface object at an appropriate position when a set of conditions hasbeen met without requiring further user input (e.g., further user inputto adjust the position of the first user interface object). Performingan operation when a set of conditions has been met without requiringfurther user input enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, detecting the at least one of movement of the firstuser (e.g., user 7002 in FIGS. 7D-7F, or another user, etc.) in thefirst physical environment (e.g., scene 105 in FIGS. 7D-7F, or anotherscene, etc.) and movement of the first object (e.g., user 7102 in FIGS.7D-7F, or another person or object, etc.) in the second physicalenvironment (e.g., scene 105-b, or another scene, etc.) includesconcurrently detecting third movement of the first user (e.g., movementof the first user 7002 along path 7300 in FIGS. 7D-7F, or movement alonganother path, etc.) in the first physical environment and fourthmovement of the first object (e.g., movement of the second user 7102 orobject represented by 7102 along path 7302 in FIGS. 7D-7F, or movementalong another path, etc.) in the second physical environment. In someembodiments, detecting the third movement of the first user in the firstphysical environment is performed using sensors collocated with thefirst user, and detecting the fourth movement of the first object isperformed, optionally, using sensors collocated with the first objectand/or in accordance with unadjusted updated positions of the first userinterface object received from another computer system. During at leastone of the third movement of the first user in the first physicalenvironment and the fourth movement of the first object in the secondphysical environment (e.g., during movement of the first object (e.g., asecond user, an animal, a flying drone, etc.) as a whole that causesmovement of the first user interface object in the three-dimensionalenvironment and movement of the first user as a whole that causesmovement of the viewpoint of the currently displayed view of thethree-dimensional environment and/or movement of the virtual position ofthe first user in the three-dimensional environment): in accordance withthe determination that the respective position of the first userinterface object in the three-dimensional environment that correspondsto the location of the first object in the second physical environmentin the first manner (e.g., a new position in the three-dimensionalenvironment that corresponds to the new location of the first object inthe second physical environment during the fourth movement of the firstobject) is more than the threshold distance from the respective positionin the three-dimensional environment that corresponds to a viewpointassociated with a currently displayed view (e.g., updated according tothe third movement of the first user in the first physical environmentin accordance with a reality-mimicking mapping relationship) of thethree-dimensional environment (e.g., the representation of the seconduser or virtual object is outside of a threshold range of the firstuser's viewpoint or virtual position in the three-dimensionalenvironment that has been updated due to the third movement of the firstuser), the first user interface object is displayed at (e.g., shown tomove to) an updated position (e.g., a position that is the respectiveposition that is determined based on the reality-mimicking mappingrelationship, a position that is outside of the threshold distance fromthe virtual representation of the first user or the viewpoint of thecurrently displayed view of the three-dimensional environment, etc.) inthe three-dimensional environment (e.g., while the viewpoint of thecurrently displayed view of the three-dimensional environment is beingupdated in accordance with the third movement of the first user),wherein the updated position in the three-dimensional environmentcorresponds to an updated location of the first object in the secondphysical environment as a result of the fourth movement in the firstmanner (e.g., the appearance and location of the first user interfaceobject vary visually to the first user due to the change in therespective three-dimensional position of the first user interface objectin the three-dimensional environment, while the viewpoint of thecurrently displayed view is updated in accordance with the thirdmovement of the first user). During at least one of the third movementof the first user in the first physical environment and the fourthmovement of the first object in the second physical environment (e.g.,during movement of the first object (e.g., a second user, an animal, aflying drone, etc.) as a whole that causes movement of the first userinterface object in the three-dimensional environment and movement ofthe first user as a whole that causes movement of the viewpoint of thecurrently displayed view of the three-dimensional environment and/ormovement of the virtual position of the first user in thethree-dimensional environment): in accordance with the determinationthat the respective position of the first user interface object in thethree-dimensional environment that corresponds to the location of thefirst object in the second physical environment in the first manner(e.g., a new position in the three-dimensional environment thatcorresponds to the new location of the first object in the secondphysical environment during the second movement of the first object)does not exceed (e.g., equal to, less than, etc.) the threshold distancefrom the respective position in the three-dimensional environment thatcorresponds to the viewpoint associated with the currently displayedview (e.g., updated according to the third movement of the first user inthe first physical environment in accordance with a reality-mimickingmapping relationship) of the three-dimensional environment (e.g., therepresentation of the second user or virtual object will be within thethreshold range of the first user's viewpoint or virtual position in thethree-dimensional environment, if not otherwise shifted away), the firstuser interface object is displayed at (e.g., shown to move to) anadjusted updated position (e.g., a position that is offset from therespective position that is determined based on the reality-mimickingmapping relationship, a position that is outside of the thresholddistance from the virtual representation of the first user or theviewpoint of the currently displayed view of the three-dimensionalenvironment, etc.) in the three-dimensional environment, wherein theadjusted updated position in the three-dimensional environmentcorresponds to the location of the first object in the second physicalenvironment in a third manner (e.g., taking into account both the fourthmovement of the first object and the third movement of the first user)different from the first manner (and, optionally, different from thesecond manner). For example, in some embodiments, when the second useror virtual object moves in a direction in the second physicalenvironment that corresponds to a movement toward the viewpoint of thecurrently displayed view or the virtual position of the first user inthe three-dimensional environment and/or the first user moves in adirection in the first physical environment that corresponds to amovement toward the representation of the second user or the virtualobject in the three-dimensional environment, the first user interfaceobject appears to be under the influence of another force that pushes itaway from and/or that prevents it from getting too close to theviewpoint or the virtual position of the first user when therepresentation of the second user or virtual object is about to crossthe threshold distance of the changing viewpoint or the virtual positionof the first user in the three-dimensional environment. Therepresentation of the second user and virtual object is restored totheir original movement path calculated using the defaultreality-mimicking mapping relationship after the condition fortriggering the collision avoidance offset is no longer met. In someembodiments, the virtual position of the first user or the viewpoint ofthe currently displayed view of the three-dimensional environmentcorresponds to the location of the first user in the first physicalenvironment in the first manner (e.g., a reality-mimicking manner, apreset first mapping relationship, etc.), irrespective of the movementand current location of the second user or virtual object in the secondphysical environment. In some embodiments, the first user interfaceobject is a floating talking avatar representing a second user that isin a virtual conference call with the first user; when the first userand the second user respectively walk within their physical environment,the virtual positions of the first user and the second user in thethree-dimensional environment are determined based on the movement ofthe first user and the second user in accordance with some presetdefault mapping relationships respectively applied to the first user andthe second user. For example, the virtual positions of the first userand the second user are confined to the same common virtual space, eventhough the first user or the second user may have executed significantmovement distances in their respective physical environments. In theevent that, the default mapping would result in a visual collision ofthe virtual positions of the first user and the second user in thethree-dimensional environment, the floating talking avatar of the seconduser (when viewed via the first display generation component by thefirst user) moves out of the way as the floating talking avatar gets tooclose to the virtual position of the first user. At the same time, thefloating talking avatar of the first user (when viewed via the seconddisplay generation component by the second user) moves out of the way asthe floating talking avatar gets too close to the virtual position ofthe second user.

Concurrently detecting third movement of the first user in the firstphysical environment and fourth movement of the first object in thesecond physical environment, and displaying the first user interfaceobject at the respective position of the first user interface object inthe three-dimensional environment that corresponds to the respectivelocation of the first object in the second physical environment in thefirst manner in accordance with the determination that the respectiveposition of the first user interface object in the three-dimensionalenvironment that corresponds to the location of the first object in thesecond physical environment in the first manner is more than thethreshold distance from the respective position in the three-dimensionalenvironment that corresponds to a viewpoint associated with a currentlydisplayed view of the three-dimensional environment, and displaying thefirst user interface object at an adjusted position in thethree-dimensional environment corresponding to the respective locationof the first object in the second physical environment in a third mannerdifferent from the first manner, in accordance with the determinationthat the respective position of the first user interface object in thethree-dimensional environment that corresponds to the location of thefirst object in the second physical environment in the first manner doesnot exceed the threshold distance from the respective position in thethree-dimensional environment that corresponds to the viewpointassociated with the currently displayed view of the three-dimensionalenvironment, displays the first user interface object at an appropriateposition when a set of conditions has been met without requiring furtheruser input (e.g., further user input to adjust the position of the firstuser interface object). Performing an operation when a set of conditionshas been met without requiring further user input enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the first object is a second user (e.g., the seconduser 7102 in FIGS. 7D-7F, or another user, etc.) that is located in thesecond physical environment (e.g., scene 105-b in FIGS. 7D-7F), and thefirst user (e.g., the first user 7002) and the second user (e.g., thesecond user 7102) at least partially shares (e.g., both users canconcurrently view, access, and/or interact with at least a portion of)the three-dimensional environment (e.g., the environment viewed via thedisplay generation components 7100 and 7200 in FIGS. 7D-7F). Forexample, the second user is provided with a view of thethree-dimensional environment via a second display generation componentin communication with a second computer system. The location of thesecond user in the second physical environment are detected by one ormore sensors that are in communication with the second computer system.In some embodiments, the second computer system sends and the firstcomputer system receives data representing the second physicalenvironment, and the first computer system and the second computersystem both display a representation of the second physical environment(e.g., a camera view or pass-through view of the second physicalenvironment). In some embodiments, the first computer system sends andthe second computer system receives data representing the first physicalenvironment, and the first computer system and the second computersystem both display a representation of the first physical environment(e.g., a camera view or pass-through view of the first physicalenvironment). In some embodiments, the first computer system and thesecond computer system both display a representation of a virtualenvironment (e.g., a virtual conference room, a gaming world, etc.) or arepresentation of a third physical environment (e.g., a physicalenvironment of a third user that is different from the first user andthe second user, and that is also sharing the three-dimensionalenvironment with the first user and the second user, etc.) differentfrom the first physical environment and the second physical environment.In some embodiments, when the display position of the second user ismodified with the visual collision avoidance shifts/offset (e.g., in thesecond manner, with the second, collision-avoidance mappingrelationship, etc.) in the view of the three-dimensional environmentprovided to the first user via the first display generation component,the view of the three-dimensional environment provided to the seconduser via a second display generation component shows instead that thedisplayed position of the first user is modified with the visualcollision avoidance shift/offset while the viewpoint of thethree-dimensional environment that is based on the second user'slocation does not have such a shift or offset.

Displaying the first user interface object at the respective position ofthe first user interface object in the three-dimensional environment, inaccordance with the determination that the respective position of thefirst user interface object in the three-dimensional environment thatcorresponds to the respective location of the first object in the secondphysical environment in the first manner is more than the thresholddistance from the respective position in the three-dimensionalenvironment that corresponds to a viewpoint associated with a currentlydisplayed view of the three-dimensional environment, and displaying thefirst user interface object at a second display position in the secondview of the three-dimensional environment offset from the respectiveposition of the first user interface object in the three-dimensionalenvironment, in accordance with the determination that the respectiveposition of the first user interface object in the three-dimensionalenvironment that corresponds to the respective location of the firstobject in the second physical environment in the first manner is notmore than the threshold distance from the respective position in thethree-dimensional environment that corresponds to the viewpointassociated with the currently displayed view of the three-dimensionalenvironment, wherein the first object is a second user that is locatedin the second physical environment, and the first user and the seconduser at least partially shares the three-dimensional environment,displays the first user interface object at an appropriate position whena set of conditions has been met without requiring further user input(e.g., further user input to adjust the position of the first userinterface object). Performing an operation when a set of conditions hasbeen met without requiring further user input enhances the operabilityof the device, which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, in a third view of the three-dimensionalenvironment (e.g., the view 7304-b shown in FIG. 7F(B)) that isdisplayed via a second display generation component (e.g., displaygeneration component 7200, or another display generation component,etc.) to the second user (e.g., user 7102, or another user, etc.), inaccordance with a determination that a respective position of a seconduser interface object (e.g., the representation 7002′-b in FIG. 7F,another representation, etc.) (e.g., an avatar of the first user, atalking-head representing the first user, etc.) in the three-dimensionalenvironment that corresponds to a respective location of the first user(e.g., user 7002 in FIGS. 7D-7F, another user, etc.) in the firstphysical environment (e.g., scene 105-a in FIGS. 7D-7F) in the firstmanner (e.g., determined in accordance with a reality-mimicking manner,with the preset first mapping relationship, etc.) does not exceed thethreshold distance from a respective position in the three-dimensionalenvironment that corresponds to a third viewpoint associated with thethird view of the three-dimensional environment (e.g., the virtualposition of the second user in the three-dimensional environment), thecomputer system displays the second user interface object (e.g.,representation 7002′-b of the first user 7002) at a modified position inthe third view of the three-dimensional environment, wherein themodified position is offset from the respective position of the seconduser interface object in the three-dimensional environment thatcorresponds to the respective location of the first user in the firstphysical environment in the first manner (e.g., determined in accordancewith a reality-mimicking manner, with the preset first mappingrelationship, etc.) (e.g., as shown in FIG. 7F(B), the representation7002′-b of the first user 7002 is displayed to the right of therepresentation 7102′-b in the view 7304-b″ shown to the second user7102, even though the position of the representation 7102′-b of thefirst user 7002 should be straight in front of the representation7102′-b in the view 7304-b″).

Displaying the second user interface object at a modified position inthe third view of the three-dimensional environment that is offset fromthe respective position of the second user interface object in thethree-dimensional environment that corresponds to the respectivelocation of the first user in the first physical environment in thefirst manner, in accordance with a determination that a respectiveposition of a second user interface object in the three-dimensionalenvironment that corresponds to a respective location of the first userin the first physical environment in the first manner does not exceedthe threshold distance from a respective position in thethree-dimensional environment that corresponds to a third viewpointassociated with the third view of the three-dimensional environment,displays the first user interface object at an appropriate position whena set of conditions has been met without requiring further user input(e.g., further user input to adjust the position of the first userinterface object). Performing an operation when a set of conditions hasbeen met without requiring further user input enhances the operabilityof the device, which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, the first object is a physical object that islocated in the second physical environment (e.g., user 7102 in FIGS.7D-7F can represents a physical object that is not another user usingthe display generation component 7200, in some embodiments). In someembodiments, the three-dimensional environment includes representationsof other physical objects in a different physical environment from thatof the viewer of the three-dimensional environment. For example, in someembodiments, the three-dimensional environment is optionally part of asimulated nature walk on a hiking trail for the first user, and includesa real-time camera feed or recorded video of the physical environment onthe hiking trail at a different physical location from the physicalenvironment of the first user. As the first user is walking in the firstphysical environment and experiencing the simulated nature walk with aview of the physical environment from the hiking trail, if a physicalobject, such as a wild bird, a flying drone, or another hiker, is alsomoving in the physical environment surrounding the hiking trail suchthat a respective position of the physical object would overlap with orbe within a threshold distance of the virtual position of the first user(e.g., a position corresponding to the viewpoint of the view of thehiking trail currently shown to the first user, or a position of avirtual representation of the first user in the view, etc.) withoutvisual collision avoidance adjustment, the experience of the first userwould be disrupted by the visual collision (e.g., the first user wouldfeel like that the physical object ran through his/her head or body,and/or would fear that such a sensation or experience would occur,etc.). By automatically adjusting the displayed position of therepresentation of the physical object in such a scenario, such that, thevisual collision would be avoided, the first user would be able toexperience the simulated nature walk more comfortably and smoothly. Inanother example, the first object is a stationary object, such as a bigrock on the hiking trail or a person sitting in the middle of the hikingtrail. In some embodiments, the first user interface object isautomatically moved out of the way (e.g., through digital image editingmeans) when the virtual position of the first user approaches theposition of the big rock or sitting person in the view of the hikingtrail. In some embodiments, in accordance with a determination that thefirst object is a stationary physical object (e.g., throughout theexperience), the first computer system optionally ceases to display thefirst user interface object briefly when it is too close to the virtualposition of the first user in the currently displayed view of thethree-dimensional environment.

Displaying the first user interface object at the respective position ofthe first user interface object in the three-dimensional environment, inaccordance with the determination that the respective position of thefirst user interface object in the three-dimensional environment thatcorresponds to the respective location of the first object in the secondphysical environment in the first manner is more than the thresholddistance from the respective position in the three-dimensionalenvironment that corresponds to a viewpoint associated with a currentlydisplayed view of the three-dimensional environment, and displaying thefirst user interface object at a second display position in the secondview of the three-dimensional environment offset from the respectiveposition of the first user interface object in the three-dimensionalenvironment, in accordance with the determination that the respectiveposition of the first user interface object in the three-dimensionalenvironment that corresponds to the respective location of the firstobject in the second physical environment in the first manner is notmore than the threshold distance from the respective position in thethree-dimensional environment that corresponds to the viewpointassociated with the currently displayed view of the three-dimensionalenvironment, wherein the first object is a physical object that islocated in the second physical environment, displays the first userinterface object at an appropriate position when a set of conditions hasbeen met without requiring further user input (e.g., further user inputto adjust the position of the first user interface object). Performingan operation when a set of conditions has been met without requiringfurther user input enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, the first user interface object (e.g.,representation 7102′-a of the second user 7102 in FIGS. 7D-7F) is anobject that is floating (e.g., unattached to and/or touching anotherobject or surface) in the three-dimensional environment (e.g., afloating talking-head avatar of the second user, a representation of aflying bird, a representation of a flying drone, etc.). In someembodiments, representations of inanimate objects in the currentlydisplayed view of the three-dimensional environment also havecorresponding visual collision-avoidance movement in a similar manner.When the first user moves to cause the virtual position corresponding tothe viewpoint of the currently displayed view of the three-dimensionalenvironment to approach a floating virtual lantern or a virtualrepresentation of a digital assistant, the floating virtual lantern orthe virtual representation of the digital assistant automatically moveout of the way to the side in the currently displayed view of thethree-dimensional environment. Displaying the first user interfaceobject at the respective position of the first user interface object inthe three-dimensional environment, in accordance with the determinationthat the respective position of the first user interface object in thethree-dimensional environment that corresponds to the respectivelocation of the first object in the second physical environment in thefirst manner is more than the threshold distance from the respectiveposition in the three-dimensional environment that corresponds to aviewpoint associated with a currently displayed view of thethree-dimensional environment, and displaying the first user interfaceobject at a second display position in the second view of thethree-dimensional environment offset from the respective position of thefirst user interface object in the three-dimensional environment, inaccordance with the determination that the respective position of thefirst user interface object in the three-dimensional environment thatcorresponds to the respective location of the first object in the secondphysical environment in the first manner is not more than the thresholddistance from the respective position in the three-dimensionalenvironment that corresponds to the viewpoint associated with thecurrently displayed view of the three-dimensional environment, whereinthe first user interface object is an object that is floating in thethree-dimensional environment, displays the first user interface objectat an appropriate position when a set of conditions has been met withoutrequiring further user input (e.g., further user input to adjust theposition of the first user interface object). Performing an operationwhen a set of conditions has been met without requiring further userinput enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, in accordance with a determination that thethree-dimensional environment is displayed with a first level ofrealism, the first user interface object (e.g., representation 7102′-aof the second user 7102 in FIGS. 7D-7F) is displayed with a first set ofdisplay properties (e.g., first resolution, first number of dimensions,first level of clarity, first color palette, without lighting effect,etc.) that corresponds to the first level of realism; and in accordancewith a determination that the three-dimensional environment is displayedwith a second level of realism that is different from (e.g., greaterthan, less than, etc.) the first level of realism, the first userinterface object is displayed with a second set of display properties(e.g., second resolution, second number of dimensions, second level ofclarity, second color palette, with lighting effect, etc.) thatcorresponds to the second level of realism, the second set of displayproperties are different from (e.g., greater than, less than, adding to,subtracting from, etc.) the first set of display properties. Forexample, in some embodiments, when the three-dimensional environment isa virtual three-dimensional environment, the first user interface objectis a three-dimensional virtual model of the second user with fixedtalking animation; and when the three-dimensional environment is anaugmented reality environment, the first user interface object is a morerealistic three-dimensional model of the second user with facialexpressions and talking animations that are generated in accordance withreal-time video image of the second user. In some embodiments, thecomputer system switches from displaying the three-dimensionalenvironment with a first level of realism to a second level of realismin response to a user's request (e.g., the first user, the second user,etc.) to switch from a virtual mode to an augmented reality mode of theshared experience. In some embodiments, the first user interface objectis a two-dimensional image of the second user floating in thethree-dimensional environment when the three-dimensional environment hassimple, flat and opaque surfaces; and the first user interface objectbecomes a three-dimensional model of the second user with more exquisitefacial features and lighting effects when the three-dimensionalenvironment is switched to a more realistic virtual rendering of aphysical environment. In some embodiments, automatically matching thelevel of realism of the first user interface object and/or how the firstuser interface object is rendered in the three-dimensional environmentto the level of realism of the three-dimensional environment and/or howthe three-dimensional environment is rendered makes the first userinterface object appear a natural, unobtrusive part of thethree-dimensional environment, thereby improving the viewing experienceof the first user.

Displaying the first user interface object with a first set of displayproperties that corresponds to a first level of realism, in accordancewith a determination that the three-dimensional environment is displayedwith the first level of realism, and displaying the first user interfaceobject with a second set of display properties, different from the firstset of display properties, that corresponds to the second level ofrealism, in accordance with a determination that the three-dimensionalenvironment is displayed with a second level of realism that isdifferent from the first level of realism, provides improved visualfeedback to the user (e.g., improved visual feedback regarding whichlevel of realism the three-dimensional environment is displayed with).Providing improved feedback enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, the second display position in the second view ofthe three-dimensional environment (e.g., the display position of therepresentation 7102′-a of the second user 7102 in view 7304-a″ shown tothe first user 7002) is displaced from the respective position of thefirst user interface object (e.g., representation 7102′-a of the seconduser 7102) in the three-dimensional environment that corresponds to therespective location of the first object (e.g., the second user 7102 inFIG. 7F, or another object, etc.) in the second physical environment(e.g., scene 105-b, or another scene, etc.) in the first manner by afirst displacement amount, wherein the first displacement amount (e.g.,having a first direction, and/or a first magnitude, etc.) does notcorrespond to movement of the first object in the second physicalenvironment (e.g., movement along path 7302 in FIGS. 7D-7F, or movementalong another path) in the first manner. In some embodiments, the firstdisplacement amount does not correspond to the movement of the firstobject in the second physical environment at all, or the firstdisplacement amount corresponds to the movement of the first object inthe second physical environment in a respective manner different fromthe first manner, etc.

Displaying the first user interface object at a second display positionin the second view of the three-dimensional environment, wherein thesecond display position in the second view of the three-dimensionalenvironment is displaced from the respective position of the first userinterface object in the three-dimensional environment that correspondsto the respective location of the first object in the second physicalenvironment in the first manner by a first displacement amount that doesnot correspond to movement of the first object in the second physicalenvironment in the first manner, in accordance with a determination thatthe respective position of the first user interface object in thethree-dimensional environment that corresponds to the respectivelocation of the first object in the second physical environment in thefirst manner is less than the threshold distance from the respectiveposition in the three-dimensional environment that corresponds to thesecond viewpoint associated with the second view of thethree-dimensional environment, provides improved visual feedback to theuser (e.g., improved visual feedback that the respective position of thefirst user interface object in the three-dimensional environment thatcorresponds to the respective location of the first object in the secondphysical environment in the first manner is less than the thresholddistance from the respective position in the three-dimensionalenvironment that corresponds to the second viewpoint associated with thesecond view of the three-dimensional environment). Providing improvedfeedback enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, the first displacement amount has a direction thatis determined in accordance with a spatial relationship (e.g., distance,and relative positions, etc.) between a viewpoint of the currentlydisplayed view of the three-dimensional environment (e.g., viewpoint ofview 7304-a″ in FIG. 7F) and the respective position of the first userinterface object (e.g., representation 7102′-a of the second user 7102in FIG. 7F, or another representation, etc.) in the three-dimensionalenvironment that corresponds to the respective location of the firstobject (e.g., second user 7102 in FIGS. 7D-7F, or another user orobject, etc.) in the second physical environment (e.g., scene 105-b, oranother scene, etc.) in the first manner.

Displaying the first user interface object at a second display positionin the second view of the three-dimensional environment, wherein thesecond display position in the second view of the three-dimensionalenvironment is displaced from the respective position of the first userinterface object in the three-dimensional environment that correspondsto the respective location of the first object in the second physicalenvironment in the first manner by a first displacement amount that hasa direction that is determined in accordance with a spatial relationshipbetween a viewpoint of the currently displayed view of thethree-dimensional environment and the respective position of the firstuser interface object in the three-dimensional environment thatcorresponds to the respective location of the first object in the secondphysical environment in the first manner, provides improved visualfeedback to the user (e.g., improved visual feedback regarding thespatial relationship between the viewpoint of the currently displayedview and the respective position of the first user interface object).Providing improved feedback enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, the second display position in the second view ofthe three-dimensional environment (e.g., view 7304-a″ in FIG. 7F(A), oranother view, etc.) is displaced from the respective position of thefirst user interface object (e.g., representation 7102′-a of the seconduser, another representation, etc.) in the three-dimensional environmentthat corresponds to the respective location of the first object (e.g.,second user 7102, another user or object, etc.) in the second physicalenvironment (e.g., scene 105-b, another scene, etc.) in the first mannerby a second displacement amount (e.g., same as the first displacementamount, different from the first displacement amount, etc.), wherein thesecond displacement amount has a direction that is different from (e.g.,perpendicular or orthogonal, to the left of, to the right of, pointingupward from, etc.) a forward direction toward the respective position inthe three-dimensional environment that corresponds to the secondviewpoint associated with the second view of the three-dimensionalenvironment (e.g., as shown in FIG. 7F(A), the representation 7102′-a isshifted to the side of the viewpoint, even though its position should beright in front of the viewpoint in the view 7304-a″ shown to the firstuser 7002). For example, in some embodiments, irrespective of from whichdirection the first user interface object is approaching and reachingthe respective position that corresponds to the viewpoint associatedwith the currently displayed view of the three-dimensional environment(e.g., the second view, or another view displayed later, etc.), thefirst user interface object is diverted to the side or above therespective position that corresponds to the viewpoint associated withthe currently displayed view, so that the displayed position of thefirst user interface object does not enter a restricted spacesurrounding the respective position corresponding to the viewpoint, evenif the respective position of the first user interface object calculatedbased on the movement of the first user and/or the first object in theirrespective physical environments in accordance with the defaultunadjusted manner (e.g., the first manner, the reality-mimicking manner,according to a first preset mapping relationship, etc.) gets even closeror passes through the respective viewpoint associated with the currentlydisplayed view of the three-dimensional environment. In someembodiments, it is as if there were an invisible glass wall at athreshold distance in front of the respective position that correspondsto the viewpoint of the currently displayed view of thethree-dimensional environment that causes the first user interfaceobject to slide to the side or upward along the invisible glass walluntil the respective position of the first user interface objectcalculated based on the current location and movement history of thefirst user and the first object in the first manner is no longer withinthe threshold distance of the respective position that corresponds tothe viewpoint of the currently displayed view of the three-dimensionalenvironment. This implementation helps to avoid the first user interfaceobject from crossing the first user's viewpoint or virtualrepresentation of the first user face on.

Displaying the first user interface object at a second display positionin the second view of the three-dimensional environment, wherein thesecond display position in the second view of the three-dimensionalenvironment is displaced from the respective position of the first userinterface object in the three-dimensional environment that correspondsto the respective location of the first object in the second physicalenvironment in the first manner by a second displacement amount, whereinthe second displacement amount has a direction that is different from aforward direction toward the respective position in thethree-dimensional environment that corresponds to the second viewpointassociated with the second view of the three-dimensional environment,provides improved visual feedback to the user (e.g., improved visualfeedback that the respective position of the first user interface objectin the three-dimensional environment that corresponds to the respectivelocation of the first object in the second physical environment in thefirst manner is less than the threshold distance from the respectiveposition in the three-dimensional environment that corresponds to thesecond viewpoint associated with the second view of thethree-dimensional environment). Providing improved feedback enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the second display position in the second view(e.g., view 7304-a″ in FIG. 7F) of the three-dimensional environment isdisplaced from the respective position of the first user interfaceobject (e.g., representation 7102′-a of the second user, anotherrepresentation, etc.) in the three-dimensional environment thatcorresponds to the respective location of the first object (e.g., user7102 in FIG. 7F, another user or object, etc.) in the second physicalenvironment (e.g., scene 105-b, or another scene, etc.) in the firstmanner by a third displacement amount (e.g., same as the firstdisplacement amount, different from the first displacement amount,etc.), wherein the third displacement amount has a direction that isdifferent from (e.g., perpendicular or orthogonal, to the left of, tothe right of, pointing upward from, etc.) a direction of approachbetween (e.g., pointing toward a common center of) the first userinterface object and the respective position in the three-dimensionalenvironment that corresponds to the second viewpoint associated with thesecond view of the three-dimensional environment (e.g., as shown in FIG.7F(A), the representation 7102′-a is shifted to the side of theviewpoint, even though it should move straight through the viewpoint inthe view 7304-a″ shown to the first user 7002). For example, in someembodiments, depending on from which direction the first user interfaceobject is approaching and reaching the respective position thatcorresponds to the viewpoint associated with the currently displayedview of the three-dimensional environment (e.g., the second view, oranother view displayed later, etc.), the third displacement amountcauses the first user interface object to be diverted to from itsdirection of approach to the side or above the respective position thatcorresponds to the viewpoint associated with the currently displayedview, so that the displayed position of the first user interface objectdoes not enter a restricted space surrounding the respective positioncorresponding to the viewpoint, even if the respective position of thefirst user interface object calculated based on the movement of thefirst user and/or the first object in their respective physicalenvironments gets even closer or passes through the respective viewpointassociated with the currently displayed view of the three-dimensionalenvironment. In some embodiments, it is as if there were an invisibleglass dome surrounding the respective position that corresponds to theviewpoint of the currently displayed view of the three-dimensionalenvironment that causes the first user interface object to slide to theside or upward along the invisible glass dome until the respectiveposition of the first user interface object calculated based on thecurrent location and movement history of the first user and the firstobject is no longer within the threshold distance of the respectiveposition that corresponds to the viewpoint of the currently displayedview of the three-dimensional environment. This implementation helps toavoid the first user interface object from crossing the first user'sviewpoint or virtual representation of the first user from anydirections (e.g., face on, from the side, etc.).

Displaying the first user interface object at a second display positionin the second view of the three-dimensional environment, wherein thesecond display position in the second view of the three-dimensionalenvironment is displaced from the respective position of the first userinterface object in the three-dimensional environment that correspondsto the respective location of the first object in the second physicalenvironment in the first manner by a third displacement amount that hasa direction that is different from a direction of approach between thefirst user interface object and the respective position in thethree-dimensional environment that corresponds to the second viewpointassociated with the second view of the three-dimensional environment,provides improved visual feedback to the user (e.g., improved visualfeedback that the respective position of the first user interface objectin the three-dimensional environment that corresponds to the respectivelocation of the first object in the second physical environment in thefirst manner is less than the threshold distance from the respectiveposition in the three-dimensional environment that corresponds to thesecond viewpoint associated with the second view of thethree-dimensional environment). Providing improved feedback enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, at least one of a magnitude and a direction of adisplacement between the second display position in the second view ofthe three-dimensional environment (e.g., view 7304-a″ in FIG. 7F, oranother view, etc.) and the respective position of the first userinterface object (e.g., representation 7102′-a in FIG. 7F, or anotherrepresentation) in the three-dimensional environment that corresponds tothe respective location of the first object (e.g., user 7102 in FIG. 7F,or another user or object, etc.) in the second physical environment inthe first manner is based on (e.g., the displacement is dynamicallyadjusted in direction and/or magnitude in accordance with a magnitudeand/or direction of a change in the magnitude and/or direction of, or inaccordance with an absolute values of a magnitude and/or direction of,etc.) a spatial relationship (e.g., relative direction and/or distance,etc.) between the respective position of the first user interface objectin the three-dimensional environment that corresponds to the respectivelocation of the first object in the second physical environment in thefirst manner and the respective position in the three-dimensionalenvironment that corresponds to the second viewpoint associated with thesecond view of the three-dimensional environment.

Displaying the first user interface object at a second display positionin the second view of the three-dimensional environment, wherein atleast one of a magnitude and a direction of a displacement between thesecond display position in the second view of the three-dimensionalenvironment and the respective position of the first user interfaceobject in the three-dimensional environment that corresponds to therespective location of the first object in the second physicalenvironment in the first manner is based on a spatial relationshipbetween the respective position of the first user interface object inthe three-dimensional environment that corresponds to the respectivelocation of the first object in the second physical environment in thefirst manner and the respective position in the three-dimensionalenvironment that corresponds to the second viewpoint associated with thesecond view of the three-dimensional environment, provides improvedvisual feedback to the user (e.g., improved visual feedback regardingthe spatial relationship between the respective position of the firstuser interface object and the respective position in thethree-dimensional environment that corresponds to the second viewpointassociated with the second view of the three-dimensional environment).Providing improved feedback enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

It should be understood that the particular order in which theoperations in FIGS. 9A-9B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 8000, 10000, 11000, and 12000) are also applicable in ananalogous manner to method 9000 described above with respect to FIGS.9A-9B. For example, the gestures, gaze inputs, physical objects, userinterface objects, controls, movements, criteria, three-dimensionalenvironment, display generation component, surface, representation ofphysical object, virtual objects, and/or animations described above withreference to method 9000 optionally have one or more of thecharacteristics of the gestures, gaze inputs, physical objects, userinterface objects, controls, movements, criteria, three-dimensionalenvironment, display generation component, surface, representation ofphysical object, virtual objects, and/or animations described hereinwith reference to other methods described herein (e.g., methods 8000,10000, 11000, and 12000). For brevity, these details are not repeatedhere.

FIG. 10 is a flowchart of a method of changing a level of immersion withwhich an environment of a computer-generated experience is displayed inaccordance with changing biometric data of a user that is received bythe computer system, in accordance with some embodiments.

In some embodiments, the method 10000 is performed at a computer system(e.g., computer system 101 in FIG. 1 ) including a display generationcomponent (e.g., display generation component 120 in FIGS. 1, 3, and 4 )(e.g., a heads-up display, a display, a touchscreen, a projector, etc.)and one or more cameras (e.g., a camera (e.g., color sensors, infraredsensors, and other depth-sensing cameras) that points downward at auser's hand or a camera that points forward from the user's head). Insome embodiments, the method 10000 is governed by instructions that arestored in a non-transitory computer-readable storage medium and that areexecuted by one or more processors of a computer system, such as the oneor more processors 202 of computer system 101 (e.g., control unit 110 inFIG. 1A). Some operations in method 10000 are, optionally, combinedand/or the order of some operations is, optionally, changed.

In some embodiments, the method 10000 is performed at a computer system(e.g., computer system 101 in FIG. 1 ) that is in communication with adisplay generation component (e.g., display generation component 120 inFIGS. 1, 3, and 4 , display generation component 7100, etc.) (e.g., aheads-up display, an HMD, a display, a touchscreen, a projector, etc.)and one or more input devices (e.g., cameras, controllers,touch-sensitive surfaces, joysticks, buttons, etc.). In someembodiments, the computer system is an integrated device with one ormore processors and memory enclosed in the same housing as the displaygeneration component and at least some of the one or more input devices.In some embodiments, the computer system includes a computing componentthat includes one or more processors and memory that is separate fromthe display generation component and/or the one or more input devices.In some embodiments, the display generation component and the one ormore input devices are integrated and enclosed in the same housing.

In the method 10000, the computer system displays (10002) a firstcomputer-generated experience (e.g., an application user interface, avirtual experience, an augmented reality experience, a mixed realityexperience, etc.) with a first level of immersion (e.g., displaying atwo-dimensional application user interface, displaying a two-dimensionalview of a three-dimensional environment, displaying a window orviewpoint into a three-dimensional environment that occupies a smallfirst portion of the field of view of the user, displaying thecomputer-generated experience with non-spatial audio, etc.) (e.g., asillustrated in FIG. 7G, where a minimal amount of virtual content of thecomputer-generated experience is displayed, and the representation ofthe physical environment dominates the view 7316). While displaying thefirst computer-generated experience with the first level of immersion,the computer system receives (10004) (e.g., in real-time, through one ormore biometric sensors (e.g., various suitable medical devices,vibration sensors, cameras, thermal sensors, chemical sensors, etc.)connected to or pointed at the first user, etc.) biometric datacorresponding to a first user (e.g., the user 7002 in FIGS. 7G-7J)(e.g., corresponding to the physiological state of the first user at afirst point or period in time). In some embodiments, the biometric datadoes not include non-transient characteristics of humans (e.g.,fingerprint, iris pattern and color, facial features, voiceprint, etc.)that do not typically change over a period of time that an average useris engaged with the computer-generated experience. In some embodiments,the biometric data includes heart rate, breathing rate, bodytemperature, serum concentration of certain chemical, medication,hormones, etc., blood pressure, brain waves, focus level, pupil size,metabolic rate, blood sugar level, one or more types of biometric datathat may vary over time during a user's engagement with thecomputer-generated experience, one or more types of biometric data thatmay vary through the user's own actions (e.g., meditation, breathingpattern change, exercise, etc., as opposed to direct interaction withuser interface elements or controls provided by the computer system)during the user's engagement with the computer-generated experience, oneor more types of composite metrics of multiple types of biometric datathat correspond to a user's mood, happiness, and/or stress level, etc.In response to receiving (10006) the biometric data corresponding to thefirst user (e.g., corresponding to the physiological state of the firstuser at the first point or period in time) (e.g., periodically receiveddata, or continuously received data, etc.) and in accordance with adetermination that the biometric data corresponding to the first user(e.g., corresponding to the physiological state of the first user at thefirst point or period in time) (e.g., the most recently receivedbiometric data, the biometric data received over a most recent timeperiod of a preset duration, etc.) meets first criteria, the computersystem displays (10008) the first computer-generated experience with asecond level of immersion (e.g., the second level of immersion providesa more immersive experience than the first level of immersion, thesecond level of immersion provides a less immersive experience than thefirst level of immersion, etc.), wherein the first computer-generatedexperience displayed with the second level of immersion occupies alarger portion of a field of view (e.g., a wider angular range in thelateral direction, a wider angular range in the vertical direction, alarger viewport size, etc.) of the first user than the firstcomputer-generated experience displayed with the first level ofimmersion (e.g., the first computer-generated experience occupying alarger portion of the field of view of the first user, optionally,provides a more immersive experience to the first user than when thefirst computer-generated experience occupies a smaller portion of thefield of view of the first user). In some embodiments the first criteriainclude: preset criteria for indicating that the first user is mentallyand emotionally ready to enter a more immersive experience, presetcriteria for indicating that the first is getting ready to exit to aless immersive experience, etc. This is illustrated in FIG. 7J, wherethe view 7334 includes virtual content of the computer-generatedexperience that occupy a greater spatial extent than the view 7316 shownin FIG. 7G, for example. In some embodiments, the computer systemdetermines that the biometric data meets the preset criteria inaccordance with a determination that the heart rate is lower than afirst threshold heart rate, the breathing rate is lower than a firstthreshold breathing rate, the blood pressure is lower than a firstthreshold blood pressure, movement of the user is below a firstthreshold amount of movement during the threshold amount of time, bodytemperature of the user is lower than a first threshold bodytemperature, a metric of stress level is below a first threshold stresslevel, a metric corresponding to user's mood indicates that the user isrelaxed and happy, etc. In some embodiments, the first level ofimmersion and the second level of immersion, optionally, differ in theamount of virtual elements present in the user's view of thecomputer-generated experience, in the number of physical surfaces thatremain visible in the computer-generated experience, in the audio outputmodes used for playing the sound effect of the computer-generatedexperience, in the level of realism depicted by the computer-generatedexperience, in the number of dimensionality depicted by thecomputer-generated experience, and/or in the number of functions andinteractions made available in the computer-generated experience, etc.In the method 10000, in response to receiving the biometric datacorresponding to the first user and in accordance with a determinationthat the biometric data corresponding to the first user does not meetthe first criteria (e.g., the heart rate is greater than the firstthreshold heart rate, the blood pressure is higher than the firstthreshold blood pressure, the movement of the user is more than thefirst threshold amount of movement during the threshold amount of time,the body temperature of the user is higher than the first threshold bodytemperature, the metric of stress level is above the first thresholdstress level, the metric corresponding to the user's mood indicates thatthe user is agitated and unhappy, etc.), the computer system continues(10010) to display the first computer-generated experience with thefirst level of immersion. This is illustrated in FIG. 7G, where thebiometric data may fluctuate below the threshold indicator 7326, and theview 7316 is maintained. In some embodiments, optionally, the firstcomputer-generated experience includes visual and audio guidance (e.g.,music, scenery, inspirational messages, guided medication recording,visual, audio, or verbal instructions on breathing, etc.) helping thefirst user to enter into a state in which the corresponding biometricdata received from the first user will meet the first criteria. Thesefeatures are illustrated in FIGS. 7G-7J, for example, where the visualbalance between virtual content and the representation of the physicalenvironment are gradually and/or abruptly changed in accordance withchanges in the biometric data corresponding to the user 7002. The visualbalance and relative visual prominence between virtual content and therepresentation of the physical environment represents a level ofimmersion with which the computer-generated experience is provided tothe user.

In some embodiments, while displaying the first computer-generatedexperience with the second level of immersion, the computer systemreceives (e.g., in real-time, through one or more biometric sensors(e.g., various suitable medical devices, vibration sensors, cameras,thermal sensors, chemical sensors, etc.) connected to or pointed at thefirst user, etc.) first updated biometric data corresponding to thefirst user (e.g., corresponding to the physiological state of the firstuser at a second point or period in time that is later than the firstpoint or period in time, after the computer system has transitioned intodisplaying the first computer-generated experience with the second levelof immersion). In some embodiments, the first updated biometric dataincludes: first updated values for the heart rate, breathing rate, bodytemperature, serum concentration of certain chemical, medication,hormones, etc., blood pressure, brain waves, focus level, pupil size,metabolic rate, blood sugar level, one or more types of biometric datathat may vary over time during a user's engagement with thecomputer-generated experience, one or more types of biometric data thatmay vary through the user's own actions (e.g., meditation, breathingpattern change, exercise, etc., as opposed to direct interaction withuser interface elements or controls provided by the computer system)during the user's engagement with the computer-generated experience, oneor more types of composite metrics of multiple types of biometric datathat correspond to a user's mood, happiness, and/or stress level, etc.,that are received after a period of time. In the method 10000, inresponse to receiving the first updated biometric data corresponding tothe first user (e.g., corresponding to the physiological state of thefirst user at the second point or period in time that is later than thefirst point or period in time) and in accordance with a determinationthat the first updated biometric data corresponding to the first user(e.g., corresponding to the physiological state of the first user at thesecond point or period in time that is later than the first point orperiod in time) meets second criteria different from (e.g., morerestrictive than, more difficult to meet, etc.) the first criteria, thecomputer system displays the first computer-generated experience with athird level of immersion (e.g., the third level of immersion provides amore immersive experience than the second level of immersion, the thirdlevel of immersion provides a less immersive experience than the secondlevel of immersion, etc.), wherein the first computer-generatedexperience displayed with the third level of immersion occupies a largerportion of the field of view of the first user than the firstcomputer-generated experience displayed with the second level ofimmersion (e.g., the first computer-generated experience occupying aneven larger portion of the field of view of the first user, optionally,provides a more immersive experience to the first user than when thefirst computer-generated experience occupies a less large portion of thefield of view of the first user). In some embodiments, the first levelof immersion, the second level of immersion, and the third level ofimmersion, optionally, differ in the amount of virtual elements presentin the user's view of the computer-generated experience, in the numberof physical surfaces that remain visible in the computer-generatedexperience, in the audio output modes used for playing the sound effectof the computer-generated experience, in the level of realism depictedby the computer-generated experience, in the number of dimensionalitydepicted by the computer-generated experience, and/or in the number offunctions and interactions made available in the computer-generatedexperience, etc. In the method 10000, in response to receiving the firstupdated biometric data corresponding to the first user and in accordancewith a determination that the first updated biometric data correspondingto the first user meets the first criteria and does not meet the secondcriteria (e.g., the heart rate is less than the first threshold heartrate but greater than the second threshold heart rate, the bloodpressure is less than the first threshold blood pressure but greaterthan the second threshold blood pressure, the movement of the user isless than the first threshold amount of movement but greater than asecond threshold amount of movement during the threshold amount of time,the body temperature of the user is less than the first threshold bodytemperature but greater than the second threshold temperature, themetric of stress level is lower than the threshold stress level butabove the second threshold stress level, the metric corresponding to theuser's mood indicates that the user is relaxed and happy but not yetfocused and peaceful, etc.), the computer system continues to displaythe first computer-generated experience with the second level ofimmersion. In some embodiments, optionally, the first computer-generatedexperience includes visual and audio guidance (e.g., music, scenery,inspirational messages, guided medication recording, visual, audio, orverbal instructions on breathing, etc.) helping the first user to enterinto a state in which the corresponding biometric data received from thefirst user will meet the second criteria. In some embodiments, thefirst, second, and third levels of immersion correspond to increasingamount of virtual content that is present in the computer-generatedenvironment and/or decreasing amount of representations of thesurrounding physical environment present in the computer-generatedenvironment. In some embodiments, first, second, and third levels ofimmersion correspond to different modes of content display that haveincreasing image fidelity and/or spatial extent (e.g., angular extent,spatial depth, etc.) for the computer-generated content, and decreasingimage fidelity and/or spatial extent for representations of thesurrounding physical environment. In some embodiments, the first levelof immersion is a pass-through mode where the physical environment isfully visible to the user through the first display generation component(e.g., as a camera view of the physical environment or through atransparent portion of the first display generation component)) and thecomputer-generated environment includes the pass-through view of thephysical environment with a minimal amount of virtual elementsconcurrently visible as the view of the physical environment orincluding virtual elements that are peripheral (e.g., indicators andcontrols displayed in the peripheral region of the display) to theuser's view of the physical environment. In some embodiments, the secondlevel of immersion is a mixed reality mode where the pass-through viewof the physical environment is augmented with virtual elements generatedby the computing system and have positions in the computer-generatedenvironment that correspond to the central portion of the user's view ofthe physical environment and/or have positions in the computer-generatedenvironment that correspond to locations and objects in the physicalenvironment (e.g., the virtual content is integrated with the physicalenvironment in the view of the computer-generated environment). In someembodiments, the third level of immersion of a virtual reality mode inwhich that user's view of the physical environment is completelyreplaced or blocked by the view of virtual content provided by the firstdisplay generation component. In some embodiments, there are fourdifferent levels of immersion, where the first level of immersioncorresponds to the pass-through mode of the first display generationcomponent, the second level of immersion includes two sub-levels A and Bthat correspond to two separate sub-modes of the first displaygeneration component (e.g., second level—A where a user interface oruser interface objects are displaying in the main portion of the user'sfield of view while the pass-through view of the physical environment isdisplayed in the background of the user interface or user interfaceobjects; and second level—B where virtual elements are integrated withrepresentations of physical objects in the physical environment in anaugmented reality view of the physical environment), and the third levelof immersion corresponds to virtual reality mode of the first displaygeneration component.

Displaying the first computer-generated experience with a third level ofimmersion that occupies a larger portion of the field of view of thefirst user than the first computer-generated experience displayed withthe second level of immersion, in accordance with a determination thatthe first updated biometric data corresponding to the first user meetssecond criteria different from the first criteria, and continuing todisplay the first computer-generated experience with the second level ofimmersion in accordance with a determination that the first updatedbiometric data corresponding to the first user meets the first criteriaand does not meet the second criteria, displays the firstcomputer-generated experience with the third level of immersion when aset of conditions has been met without requiring further user input(e.g., further user input to change the level of immersion). Performingan operation when a set of conditions has been met without requiringfurther user input enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, while displaying the first computer-generatedexperience with a respective level of immersion (e.g., the second levelof immersion, the third level of immersion, etc.), the computer systemreceives second updated biometric data corresponding to the first user(e.g., corresponding to the physiological state of the first user at athird point or period in time that is later than the first point orperiod in time and/or the second point or period in time, after thecomputer system has transitioned into displaying the firstcomputer-generated experience with the respective level of immersionfrom another, less immersive, level of immersion), wherein the firstcomputer-generated experience displayed with respective level ofimmersion occupies a larger portion of the field of view of the firstuser than the first level of immersion (e.g., the respective level ofimmersion is the second level of immersion, or the third level ofimmersion). In some embodiments, the second updated biometric dataincludes second updated values for the heart rate, breathing rate, bodytemperature, serum concentration of certain chemical, medication,hormones, etc., blood pressure, brain waves, focus level, pupil size,metabolic rate, blood sugar level, one or more types of biometric datathat may vary over time during a user's engagement with thecomputer-generated experience, one or more types of biometric data thatmay vary through the user's own actions (e.g., meditation, breathingpattern change, exercise, etc., as opposed to direct interaction withuser interface elements or controls provided by the computer system)during the user's engagement with the computer-generated experience, oneor more types of composite metrics of multiple types of biometric datathat correspond to a user's mood, happiness, and/or stress level, etc.,that are received after a period of time. In response to receiving thesecond updated biometric data corresponding to the first user (e.g.,corresponding to the physiological state of the first user at the thirdpoint or period in time that is later than the first point or period intime and/or the second point or period in time) and in accordance with adetermination that the second updated biometric data corresponding tothe first user (e.g., corresponding to the physiological state of thefirst user at the second point or period in time that is later than thefirst point or period in time) does not meet respective criteria (e.g.,the first criteria, the second criteria, etc.) that were met totransition into displaying the first computer-generated experience withthe respective level of immersion (e.g., the second level of immersion,the third level of immersion, etc.), the computer system displays thefirst computer-generated experience with a lower level of immersion(e.g., the first level of immersion, the second level of immersion,etc.) that is used prior to displaying the first computer-generatedexperience with the respective level of immersion (e.g., the secondlevel of immersion, the third level of immersion, etc.). In someembodiments, changing the level of immersion of the computer-generatedenvironment displayed via the first display generation componentincludes: in accordance with a determination that the currently receivedbiometric data no longer meets the second criteria but still meets thefirst criteria, switching from displaying the computer-generatedenvironment with the third level of immersion (e.g., virtual realitymode) to displaying the computer-generated environment with the secondlevel of immersion (e.g., a mixed reality mode, or a temporarypass-through mode optionally with concurrent display of the virtualreality content). In some embodiments, when the computer-generatedenvironment is currently displayed with the second level of immersion,and the computer system detects that the current biometric data nolonger meets the first criteria and does not meet the second criteria,the computing system switches from displaying the computer-generatedenvironment with the second level of immersion to displaying thecomputer-generated environment with the first level of immersion (e.g.,switching from the mixed reality mode (e.g., the sub-mode A of the mixedreality mode) to the complete pass-through mode, or causing display of agraphical user interface (e.g., a home screen, an application launchinguser interface) or user interface objects (e.g., application launchicons, representations of content items and experiences, etc.) to bedisplayed in the main portion of the user's field of view). For example,in FIG. 7J, after the computer-generated experience is displayed with ahigh level of immersion in response to the biometric data 7312 meetingthe preset threshold indicated by indicator 7326, the computer systemoptionally returns to any of the states shown in FIGS. 7G-7I, when thebiometric data no longer meets the preset threshold, in accordance withsome embodiments.

Displaying the first computer-generated experience with a lower level ofimmersion that is used prior to displaying the first computer-generatedexperience with the respective level of immersion, in accordance with adetermination that the second updated biometric data corresponding tothe first user does not meet respective criteria that were met totransition into displaying the first computer-generated experience withthe respective level of immersion, displays the first computer-generatedexperience with the appropriate level of immersion when as set ofconditions has been met without requiring further user input (e.g.,further user input to select the level of immersion). Performing anoperation when a set of conditions has been met without requiringfurther user input enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, the biometric data (e.g., biometric data 7312 inFIGS. 7G-7J) includes a respiration rate of the first user and the firstcriteria include a criterion that is met when the respiration rate ofthe first user is below a first threshold respiration rate in order forthe first criteria to be met. In some embodiments, the biometric datasuch as the respiration rate is used as an indication of whether thefirst user is ready to enter a deeper immersive experience provided bythe computer system and receive fewer stimuli from the physicalenvironment surrounding the first user. In some embodiments, a lowerrespiration rate, optionally, in combination with other types ofbiometric data, is used to indicate that the user is getting ready tomove to the next stage of a guided meditation provided by thecomputer-generated experience. In some embodiments, the biometric datainclude other types of physiological data, and the first criteriainclude respective threshold values for respective ones of the othertypes of physiological data. In some embodiments, the respectivethreshold values for at least a threshold number of biometric data typeshave to be met in order for the first criteria to be met. In someembodiments, the second criteria include a criterion that is met whenthe respiration rate of the first user is below a second thresholdrespiration rate that is lower than the first respiration rate in orderfor the second criteria to be met. In some embodiments, different, notnecessarily lower, values for the different types of biometric data areused for the thresholds in the second criteria.

Displaying the first computer-generated experience with a second levelof immersion in accordance with a determination that the biometric data,including the respiration rate of the first user, corresponding to thefirst user meets first criteria requiring the respiration rate of thefirst user be below a first threshold respiration rate, and continuingto display the first computer-generated experience with the first levelof immersion in accordance with a determination that the biometric data,including the respiration rate of the first user, corresponding to thefirst user does not meet the first criteria requiring the respirationrate of the first user be below a first threshold respiration rate,displays the first computer-generated experience with the appropriatelevel of immersion when a set of conditions has been met withoutrequiring further user input (e.g., further user input to select thelevel of immersion). Performing an operation when a set of conditionshas been met without requiring further user input enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the first criteria include a requirement that thebiometric data (e.g., biometric data 7312 in FIGS. 7G-7J, otherbiometric data, etc.) satisfy one or more preset threshold values (e.g.,threshold indicated by indicator 7326, other thresholds, etc.) for atleast a threshold amount of time in order for the first criteria to bemet. For example, in some embodiments, the biometric data includes arespiration rate of the first user and/or a heart rate of the firstuser, and the first criteria are met when the average respiration rateof the first user has remained below 15 breaths per minute for at leastthree minutes and/or the average heart rate of the first user hasremained below 65 beats per minute for at least five minutes. In anotherexample, the biometric data includes a blood pressure of the first userand/or an oxygenation level of the first user, and the first criteriaare met when the average blood pressure of the first user has remainedwith a first range (e.g., +/−10) of 120/80 for at least ten minutesand/or the oxygenation level of the first user has remained above 99.9%for at least three minutes.

Displaying the first computer-generated experience with a second levelof immersion in accordance with a determination that the biometric datacorresponding to the first user meets first criteria requiring thebiometric data satisfy one or more preset threshold values for at leasta threshold amount of time, and continuing to display the firstcomputer-generated experience with the first level of immersion inaccordance with a determination that the biometric data, including therespiration rate of the first user, corresponding to the first user doesnot meet the first criteria requiring the biometric data satisfy one ormore preset threshold values for at least a threshold amount of time,displays the first computer-generated experience with the appropriatelevel of immersion when a set of conditions has been met withoutrequiring further user input (e.g., further user input to select thelevel of immersion). Performing an operation when a set of conditionshas been met without requiring further user input enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, displaying the first computer-generated experiencewith the first level of immersion includes displaying virtual content(e.g., virtual content of the first computer-generated experience that,optionally, is changing over time) at respective first positions thatcorrespond to locations of one or more first portions of a physicalenvironment (e.g., the virtual content overlays, replaces display of, orblocking a view of, etc. a representation of the first portions of thephysical environment (e.g., a single continuous portion, or multipleseparate, disjointed portions, etc.) that would have been in the user'sfield of view if the virtual content were not displayed) (e.g.,displaying an augmented reality view of the physical environment, ordisplaying complete pass-through view of the physical environment with afew user interface objects, etc.), while maintaining display of (e.g.,at respective second positions) a representation of one or more secondportions (different from the first portions) of the physical environment(e.g., portions of the physical environment remain visible (e.g.,adjacent to the virtual content, as surrounding background to thevirtual content, etc.) to the user through the display generationcomponent). In some embodiments, displaying the first computer-generatedexperience with the first level of immersion includes displaying virtualcontent in a virtual window or screen that is overlaid on, replacesdisplay of, or blocking a view of, etc. a representation of a physicalenvironment (e.g., a camera view, a pass-through view through atransparent display, etc.). In some embodiments, displaying the firstcomputer-generated experience with the first level of immersion includesdisplaying virtual content at positions that correspond to a location ofa first physical surface (e.g., a real window, a wall, a tabletop, etc.)or a first number of (e.g., less than all) physical surfaces (e.g., allthe walls but not the ceiling and floor; all the walls, ceiling, andfloor, but not furniture; tabletop but not walls, etc.) in the physicalenvironment. Displaying the first computer-generated experience with thesecond level of immersion includes displaying virtual content (e.g.,virtual content of the first computer-generated experience that,optionally, is changing overtime) at the respective first positions thatcorrespond to the locations of the one or more first portions (e.g.,portions near the center of the user's field of view) of the physicalenvironment and at respective second positions that correspond to atleast some of the one or more second portions (e.g., portions fartheraway from the center of the user's field of view) of the physicalenvironment (e.g., fewer portions of the physical environment remainvisible to the user through the display generation component with thesecond level of immersion). In some embodiments, displaying the firstcomputer-generated experience with the second level of immersionincludes displaying virtual content in a three-dimensional environmentwith virtual objects that are overlaid on, replace display of, or blocka view of, etc. more or wider portions of a representation of a physicalenvironment (e.g., a camera view, a pass-through view through atransparent display, etc.). In some embodiments, displaying the firstcomputer-generated experience with the second level of immersionincludes displaying virtual content at positions that correspond tolocations of more physical surfaces and/or more types of physicalsurfaces (e.g., real window, wall, tabletop, furniture, etc.) in thephysical environment. In some embodiments, displaying the firstcomputer-generated experience with the third level of immersion includesdisplaying a virtual environment without displaying a representation ofany portion of the physical environment (e.g., displaying a virtualreality environment). In some embodiments, the virtual environment stillcorresponds to the physical environment, e.g., locations and spatialrelationships of virtual objects and surfaces in the virtual environmentstill correspond to locations and spatial relationships of at least somephysical objects and surfaces in the physical environment. In someembodiments, the virtual environment does not correspond to the physicalenvironment, except to a minimum extent (e.g., direction of gravity andorientation of the floor, etc.). In some embodiments, the firstcomputer-generated experience displayed with the first level ofimmersion is an augmented reality experience and the firstcomputer-generated experience displayed with the second level ofimmersion is a virtual experience.

Displaying the first computer-generated experience with a first level ofimmersion, including displaying virtual content at respective firstpositions that correspond to locations of one or more first portions ofa physical environment, while maintaining display of a representation ofone or more second portions of the physical environment, and displayingthe first computer-generated experience with the second level ofimmersion, including displaying virtual content at the respective firstpositions that correspond to the locations of the one or more firstportions of the physical environment and at respective second positionsthat correspond to at least some of the one or more second portions ofthe physical environment, provides improved visual feedback to the user(e.g., improved visual feedback regarding the current level ofimmersion). Providing improved feedback enhances the operability of thedevice, which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, in response to receiving the biometric data (e.g.,biometric data 7312 in FIGS. 7G-7J, or other biometric data)corresponding to the first user (e.g., corresponding to thephysiological state of the first user at the first point or period intime) (e.g., periodically received data, or continuously received data,etc.) and in accordance with a determination that a change in thebiometric data corresponding to the first user (e.g., corresponding tothe physiological state of the first user at the first point or periodin time) (e.g., the most recently received biometric data, the biometricdata received over a most recent time period of a preset duration, etc.)is progressing toward meeting the first criteria (e.g., the heart rateis slowing down to approach the first threshold heart rate, thebreathing rate is slowing down to approach the first threshold breathingrate, the body temperature is reducing to approach the first thresholdbody temperature, the serum concentration of certain chemical,medication, hormones, etc., blood pressure, brain waves, focus level,pupil size, metabolic rate, blood sugar level, one or more types ofbiometric data that may vary over time during a user's engagement withthe computer-generated experience, one or more types of biometric datathat may vary through the user's own actions (e.g., meditation,breathing pattern change, exercise, etc., as opposed to directinteraction with user interface elements or controls provided by thecomputer system) during the user's engagement with thecomputer-generated experience, one or more types of composite metrics ofmultiple types of biometric data that correspond to a user's mood,happiness, and/or stress level, etc., are changing with a trend that, ifcontinued, will cause the first criteria to be met), the computer systemgradually reduces visual emphasis of (e.g., blurring, darkening,blocking, replacing, overlaying, etc.) at least a portion of arepresentation of a physical environment that had been visible via thefirst display generation component while the first computer-generatedexperience was displayed with the first level of immersion, whereindisplaying the first computer-generated experience with the second levelof immersion includes displaying virtual content of the firstcomputer-generated experience at a position corresponding to the portionof the representation of the physical environment such that the portionof the representation of the physical environment ceases to be visiblevia the first display generation component. For example, in someembodiments, when the first computer-generated experience is displayedwith the first level of immersion, a representation of a physical wallfacing the first user (e.g., a pass-through view or camera view of thewall) is blocked, replaced, or overlaid by a virtual wall (e.g., withvirtual wallpaper), a virtual window (e.g., with virtual view), virtualscenery (e.g., an open ocean view, an open landscape, etc.), a virtualdesktop, a virtual movie screen, etc., while other physical walls,ceiling, floor, furniture in the room are still visible to the userthrough the display generation component. When the biometric datareceived from the first user meets the first criteria, the computersystem gradually blurs out and/or darkens the portions of therepresentation of the physical environment that are still visible, andreplaces them with virtual content (e.g., expansion of the existingvirtual content, adding new virtual content, etc.). In some embodiments,the computer system displays the virtual content, such as virtualwallpaper, virtual room decor, virtual scenery, virtual movie screen,virtual desktop, etc., which gradually replaces the blurred and/ordarkened portions of the representation of the physical environment(e.g., fading in from behind the portions of the representation of thephysical environment, or creeping in from surrounding regions of theportions of the representation of the physical environment, etc.). Whenthe transition is completed, the user's field of view of the firstcomputer-generated experience has been expanded and less of the physicalenvironment is visible via the display generation component.

Gradually reducing visual emphasis of at least a portion of arepresentation of a physical environment that had been visible via thefirst display generation component while the first computer-generatedexperience was displayed with the first level of immersion, anddisplaying the first computer-generated experience with the second levelof immersion, including displaying virtual content of the firstcomputer-generated experience at a position corresponding to the portionof the representation of the physical environment such that the portionof the representation of the physical environment ceases to be visiblevia the first display generation component, in accordance with adetermination that a change in the biometric data corresponding to thefirst user is progressing toward meeting the first criteria, providesimproved visual feedback to the user (e.g., improved visual feedbackthat the biometric data corresponding to the first user is progressingtowards meeting the first criteria, improved visual feedback regardingthe relative progress of the biometric data corresponding to the firstuser towards meeting the first criteria, etc.). Providing improvedfeedback enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, in response to receiving the biometric data (e.g.,biometric data 7312 in FIGS. 7G-7J, or other biometric data, etc.)corresponding to the first user (e.g., corresponding to thephysiological state of the first user at the first point or period intime) (e.g., periodically received data, or continuously received data,etc.) and in accordance with a determination that a change in thebiometric data corresponding to the first user (e.g., corresponding tothe physiological state of the first user at the first point or periodin time) (e.g., the most recently received biometric data, the biometricdata received over a most recent time period of a preset duration, etc.)is progressing toward meeting the first criteria (e.g., the heart rateis slowing down to approach the first threshold heart rate, thebreathing rate is slowing down to approach the first threshold breathingrate, the body temperature is reducing to approach the first thresholdbody temperature, the serum concentration of certain chemical,medication, hormones, etc., blood pressure, brain waves, focus level,pupil size, metabolic rate, blood sugar level, one or more types ofbiometric data that may vary over time during a user's engagement withthe computer-generated experience, one or more types of biometric datathat may vary through the user's own actions (e.g., meditation,breathing pattern change, exercise, etc., as opposed to directinteraction with user interface elements or controls provided by thecomputer system) during the user's engagement with thecomputer-generated experience, one or more types of composite metrics ofmultiple types of biometric data that correspond to a user's mood,happiness, and/or stress level, etc., are changing with a trend that, ifcontinued, will cause the first criteria to be met), the computer systemchanges a visual property of (e.g., blurring, darkening, blocking,replacing, overlaying, etc.) at least a portion of a representation of aphysical environment that had been visible via the first displaygeneration component while the first computer-generated experience wasdisplayed with the first level of immersion by an amount thatcorresponds to the change in the biometric data corresponding to thefirst user. For example, in some embodiments, when the firstcomputer-generated experience is displayed with the first level ofimmersion, a representation of a physical wall facing the first user(e.g., a pass-through view or camera view of the wall) is blocked,replaced, or overlaid by a virtual wall (e.g., with virtual wallpaper),a virtual window (e.g., with virtual view), virtual scenery (e.g., anopen ocean view, an open landscape, etc.), a virtual desktop, a virtualmovie screen, etc., while other physical walls, ceiling, floor,furniture in the room are still visible to the user through the displaygeneration component. When the biometric data received from the firstuser changes with a trend toward meeting the first criteria, thecomputer system gradually intensifies the amount of blurring and/ordarkening applied to the area of the user's field of view that are notyet covered by virtual content. Optionally, if the biometric datachanges with an opposite trend, the amount of blurring and/or darkeningis gradually reduced and the clarity of the view of the physicalenvironment in those areas improves again.

Changing a visual property of at least a portion of a representation ofa physical environment that had been visible via the first displaygeneration component while the first computer-generated experience wasdisplayed with the first level of immersion by an amount thatcorresponds to the change in the biometric data corresponding to thefirst user, in accordance with a determination that a change in thebiometric data corresponding to the first user is progressing towardmeeting the first criteria, provides improved visual feedback to theuser (e.g., improved visual feedback that the biometric datacorresponding to the first user is progressing toward meeting the firstcriteria, improved visual feedback regarding the relative progress ofthe biometric data corresponding to the first user towards meeting thefirst criteria, etc.). Providing improved feedback enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, in response to receiving the biometric data (e.g.,biometric data 7312 in FIGS. 7G-7J, or other biometric data, etc.)corresponding to the first user (e.g., corresponding to thephysiological state of the first user at the first point or period intime) (e.g., periodically received data, or continuously received data,etc.) and in accordance with a determination that a change in thebiometric data corresponding to the first user (e.g., corresponding tothe physiological state of the first user at the first point or periodin time) (e.g., the most recently received biometric data, the biometricdata received over a most recent time period of a preset duration, etc.)is progressing toward meeting the first criteria (e.g., the heart rateis slowing down to approach the first threshold heart rate, thebreathing rate is slowing down to approach the first threshold breathingrate, the body temperature is reducing to approach the first thresholdbody temperature, the serum concentration of certain chemical,medication, hormones, etc., blood pressure, brain waves, focus level,pupil size, metabolic rate, blood sugar level, one or more types ofbiometric data that may vary over time during a user's engagement withthe computer-generated experience, one or more types of biometric datathat may vary through the user's own actions (e.g., meditation,breathing pattern change, exercise, etc., as opposed to directinteraction with user interface elements or controls provided by thecomputer system) during the user's engagement with thecomputer-generated experience, one or more types of composite metrics ofmultiple types of biometric data that correspond to a user's mood,happiness, and/or stress level, etc., are changing with a trend that, ifcontinued, will cause the first criteria to be met), the computer systemexpands display of virtual content onto (e.g., blocking, replacing,overlaying, etc.) at least a portion of a representation of a physicalenvironment that had been visible via the first display generationcomponent while the first computer-generated experience was displayedwith the first level of immersion by an amount that corresponds to thechange in the biometric data corresponding to the first user. Forexample, in some embodiments, when the first computer-generatedexperience is displayed with the first level of immersion, arepresentation of a physical wall facing the first user (e.g., apass-through view or camera view of the wall) is blocked, replaced, oroverlaid by a virtual wall (e.g., with virtual wallpaper), a virtualwindow (e.g., with virtual view), virtual scenery (e.g., an open oceanview, an open landscape, etc.), a virtual desktop, a virtual moviescreen, etc., while representations of other physical walls, ceiling,floor, furniture in the room are still displayed to the user via thedisplay generation component. When the biometric data received from thefirst user changes with a trend toward meeting the first criteria, thecomputer system gradually expands the area of the user's field of viewthat is covered by virtual content to block more of the view of thesurrounding physical environment. Optionally, if the biometric datachanges with an opposite trend, the previously blocked/covered area isgradually restored and revealing the view of the physical environment inthose areas again.

Expanding display of virtual content onto at least a portion of arepresentation of a physical environment that had been visible via thefirst display generation component while the first computer-generatedexperience was displayed with the first level of immersion by an amountthat corresponds to the change in the biometric data corresponding tothe first user, in accordance with a determination that a change in thebiometric data corresponding to the first user is progressing towardmeeting the first criteria, provides improved visual feedback to theuser (e.g., improved visual feedback that the biometric datacorresponding to the first user is progressing toward meeting the firstcriteria, improved visual feedback regarding the relative progress ofthe biometric data corresponding to the first user towards meeting thefirst criteria, etc.). Providing improved feedback enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the first criteria include a criterion that thefirst user (e.g., user 7002 in FIGS. 7G-7J, or another user, etc.) makesless than a threshold amount of movement of a first type (e.g., lessthan the preset threshold amount of movement of the first type during athreshold amount of time, less than a threshold cumulative amount ofmovement of the first type, less than an absolute amount of movement ofthe first type, etc.) (e.g., movement of the first type includesmovement of the head, movement of the center of the body, movement oflimbs, and/or movement of the eyes, etc.) when the biometric data (e.g.,biometric data 7312 in FIGS. 7G-7J, or other biometric data, etc.) isbeing received in order for the first criteria to be met. For example,in some embodiments, in order ensure that the biometric data that isreceived is valid and/or ensure that the first user intends to settledown to enter into a more immersive level of the computer-generatedexperience, the first user is required to remain substantially stillwhen the biometric data is received and evaluated. In some embodiments,if more than the threshold amount of movement of the first user isdetected during the threshold amount of time, the computer system doesnot (e.g., forgoes, ceases, reverses, etc.) display of the firstcomputer-generated experience with the second level of immersion (oranother next level of immersion) even if the biometric data meets therequirements specified for the biometric data (e.g., the thresholdvalues for breathing rate, heart rate, etc.) in the first criteria.

Displaying the first computer-generated experience with a second levelof immersion in accordance with a determination that the biometric datacorresponding to the first user meets first criteria requiring that thefirst user makes less than a threshold amount of movement of a firsttype when the biometric data is being received, and continuing todisplay the first computer-generated experience with the first level ofimmersion in accordance with a determination that the biometric data,including the respiration rate of the first user, corresponding to thefirst user does not meet the first criteria requiring that the firstuser makes less than a threshold amount of movement of a first type whenthe biometric data is being received, displays the firstcomputer-generated experience with the appropriate level of immersionwhen a set of conditions has been met without requiring further userinput (e.g., further user input to select the level of immersion).Performing an operation when a set of conditions has been met withoutrequiring further user input enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, while displaying the first computer-generatedexperience with the second level of immersion, the computer systemdetects movement of a first type (e.g., movement of the head, movementof the center of the body, movement of limbs, movement of the eyes,etc.) being performed by the first user (e.g., user 7002 in FIGS. 7G-7J,another user, etc.). In response to detecting the movement of the firsttype being performed by the first user and in accordance with adetermination that the movement of the first type exceeds a presetthreshold amount of movement (e.g., more than the preset thresholdamount of movement during a threshold amount of time, more than anaccumulative amount movement, more than an absolute amount of movement,etc.), the computer system redisplays the first computer-generatedexperience with the first level of immersion. For example, in someembodiments, after the biometric data received from the first user metthe first criteria and while the first computer-generated experience isdisplayed with the second level of immersion, if the first user moves bymore than a threshold amount in one or more preset ways (e.g., the firstuser stood up, moves his/her head, stretches his arms, moved his gaze,etc.), the computer system interprets the first user's movement as anintention to exit the more immersive experience, and returns to apreviously displayed, less immersive level of the computer-generatedexperience. This feature is useful when the first user is using thecomputer-generated experience for meditation or sleep, and movement ofthe first user allows the user to return to normal state. In someembodiments, in order ensure that the biometric data that is received isvalid and/or ensure that the first user intends to settle down to enterinto a more immersive level of the computer-generated experience, thefirst user is required to remain substantially still when the biometricdata is received and evaluated. In some embodiments, if more than thethreshold amount of movement of the first user is detected during thethreshold amount of time, the computer system ceases or reverses displayof the first computer-generated experience with the second level ofimmersion (or whatever the next level of immersion is), irrespective ofwhether the biometric data still meets the requirements specified forthe biometric data (e.g., the threshold values for breathing rate, heartrate, etc.) in the first criteria.

Redisplaying the first computer-generated experience with the firstlevel of immersion in response to detecting movement of a first typebeing performed by the first user, and in accordance with adetermination that the movement of the first type exceeds a presetthreshold amount of movement, redisplays the first computer-generatedexperience with the first level of immersion when a set of conditionshas been met without requiring further user input (e.g., further userinput to select the first level of immersion). Performing an operationwhen a set of conditions has been met without requiring further userinput enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, while displaying the first computer-generatedexperience with the second level of immersion (e.g., as shown in FIG.7J), the computer system detects movement of a first type (e.g.,movement of the head, movement of the center of the body, movement oflimbs, movement of the eyes, etc.) being performed by the first user(e.g., user 7002 in FIG. 7J). In response to detecting the movement ofthe first type being performed by the first user and in accordance witha determination that the movement of the first type exceeds a presetthreshold amount of movement (e.g., more than the preset thresholdamount of movement during a threshold amount of time, more than anaccumulative amount movement, more than an absolute amount of movement,etc.), the computer system switches from displaying the firstcomputer-generated experience with the second level of immersion with afirst viewpoint to displaying the first computer-generated experiencewith the second level of immersion with a second viewpoint differentfrom the first viewpoint (e.g., the change in the viewpoint of the firstcomputer-generated experience with the second level of immersioncorresponds to the movement of the first type that is performed by thefirst user). For example, once the more immersive experience has beentriggered by the change in the biometric data, the first user can movearound in the physical environment, turn his/her head, or gaze atdifferent directions, to change the viewpoint from which the view of thevirtual environment is displayed.

Switching from displaying the first computer-generated experience withthe second level of immersion with a first viewpoint to displaying thefirst computer-generated experience with the second level of immersionwith a second viewpoint different from the first viewpoint, in responseto detecting movement of a first type being performed by the first userand in accordance with a determination that the movement of the firsttype exceeds a preset threshold amount of movement, switches thedisplayed viewpoint when a set of conditions has been met withoutrequiring further user input (e.g., further user input to change fromthe first to the second viewpoint). Performing an operation when a setof conditions has been met without requiring further user input enhancesthe operability of the device, which, additionally, reduces power usageand improves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, a transition from displaying the firstcomputer-generated experience (e.g., the computer-generated experienceshown via the first display generation component 7100 in FIGS. 7G-7J,another computer-generated experience, etc.) with the first level ofimmersion to displaying the first computer-generated experience with thesecond level of immersion is a discrete transition (e.g., abrupt changesthat simultaneously replace, block the view of, and/or overlaying on,large portions of the representation of the physical environment withvirtual content, without gradual blurring or fading in, withoutincremental changes along one or more directions across positionscorresponding to physical surfaces, etc.) that is made at a point intime that corresponds to a time that the first criteria are met. Forexample, in some embodiments, the first computer-generated experience isdisplayed with the first level of immersion for an extended period oftime before the first criteria are met, and there is a clear and abruptvisual change that is shown when the first computer-generated experiencedisplayed with the second level of immersion replaces the firstcomputer-generated experience displayed with the first level ofimmersion, upon the first criteria being met by the biometric data.

Transitioning from displaying the first computer-generated experiencewith the first level of immersion to displaying the firstcomputer-generated experience with the second level of immersion with adiscrete transition that is made at a point in time that corresponds toa time that the first criteria are met provides improved visual feedbackto the user (e.g., improved visual feedback that the computer system hastransitioned from the first level of immersion to the second level ofimmersion, improved visual feedback that the first criteria has beenmet, etc.). Providing improved feedback enhances the operability of thedevice, which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, the first computer-generated experience displayedwith the first level of immersion depicts a first virtual environmentand the first computer-generated experience displayed with the secondlevel of immersion depicts a second virtual environment that has morevirtual depth than the first virtual environment (e.g., the firstvirtual environment has virtual content on a flat, two-dimensional,surface; and the second virtual environment has virtual content atdifferent depths from the first user's viewpoint.). Displaying the firstcomputer-generated experience with the first level of immersion thatdepicts a first virtual environment, and displaying the firstcomputer-generated experience with the second level of immersion thatdepicts a second virtual environment that has more virtual depth thanthe first virtual environment, provides improved visual feedback to theuser (e.g., improved visual feedback regarding whether the computersystem is displaying the first or second level of immersion). Providingimproved feedback enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, displaying the first computer-generated experiencewith the first level of immersion includes displaying the firstcomputer-generated experience with at least a first visualcharacteristic (e.g., movement of a first virtual object, changes inlighting, etc.) that changes in accordance with a change in thebiometric data received while displaying the first computer-generatedexperience with the first level of immersion, and displaying the firstcomputer-generated experience with the second level of immersionincludes displaying the first computer-generated experience with atleast a second visual characteristic (e.g., movement of the firstvirtual object, changes in lighting, etc.) that changes in accordancewith a change in the biometric data received while displaying the firstcomputer-generated experience with the second level of immersion. Forexample, in some embodiments, the first computer-generated experiencedisplayed with the first level of immersion shows a viewport into avirtual forest night scene, virtual trees are dimly illuminated by themoon and stars on a dark virtual sky. In accordance with a change in thebiometric data received from the first user, such as a decrease inbreathing rate and/or an increase in oxygenation level, the illuminationlevel shown in the virtual forest increases accordingly, and the virtualdark sky gradually turns brighter and redder simulating arrival of dawn.When the first criteria are met by the biometric data, the firstcomputer-generated experience displayed with the second level ofimmersion shows an expanded area in the user's field of view beingoccupied by the virtual forest (e.g., the virtual forest expands aroundthe user, and surrounds the viewpoint corresponding to the currentlydisplayed view of the three-dimensional environment), and the day breaksin the virtual scene with the edge of the sun visible on the virtualhorizon. In accordance with further changes in the biometric datareceived from the first user, such as a continued decrease in breathingrate (e.g., down to a threshold level) and/or a continued increase inoxygenation level (e.g., up to a threshold level), the illuminationlevel shown in the virtual forest continues to increase accordingly, andthe virtual sky gradually turns brighter simulating arrival of daytime.In another example, the first computer-generated experience displayedwith the first level of immersion shows a virtual ocean view withcrashing waves at a position in an augmented reality environment thatcorresponds to a location of a first physical wall surface in front ofthe first user. In accordance with a change in the biometric datareceived from the first user, such as a decrease in breathing rateand/or a decrease in heart rate, the frequency and/or magnitude of theocean waves decrease accordingly. When the first criteria are met by thebiometric data, the first computer-generated experience displayed withthe second level of immersion shows an expanded area in the user's fieldof view being occupied by the ocean scene (e.g., the virtual ocean viewextends to positions that corresponds to the locations of two side wallsas well). In accordance with further changes in the biometric datareceived from the first user, such as a continued decrease in breathingrate (e.g., down to a threshold level) and/or a continued decrease inheart rate (e.g., down to a threshold level), the frequency and/ormagnitude of the virtual ocean waves continue to decrease accordingly.

Displaying the first computer-generated experience with the first levelof immersion, including displaying the first computer-generatedexperience with at least a first visual characteristic that changes inaccordance with a change in the biometric data received while displayingthe first computer-generated experience with the first level ofimmersion, and displaying the first computer-generated experience withthe second level of immersion, including displaying the firstcomputer-generated experience with at least a second visualcharacteristic that changes in accordance with a change in the biometricdata received while displaying the first computer-generated experiencewith the second level of immersion, provides improved visual feedback tothe user (e.g., improved visual feedback regarding whether the computersystem is displaying the first or second level of immersion, improvedvisual feedback regarding changes in the biometric data corresponding tothe first user, etc.). Providing improved feedback enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, in response to receiving the biometric data (e.g.,biometric data 7312 in FIGS. 7G-7J, other biometric data, etc.)corresponding to the first user (e.g., corresponding to thephysiological state of the first user (e.g., user 7002 in FIGS. 7G-7J,another user, etc.) at the first point or period in time) (e.g.,periodically received data, or continuously received data, etc.), and inaccordance with a determination that the biometric data corresponding tothe first user (e.g., corresponding to the physiological state of thefirst user at the first point or period in time) (e.g., the mostrecently received biometric data, the biometric data received over amost recent time period of a preset duration, etc.) meets the firstcriteria (e.g., as shown in FIG. 7J, the biometric data meets thethreshold indicated by indicator 7326), the computer system changes anaudio output mode from a first audio output mode to a second audiooutput mode (e.g., from stereo sound to surround sound, from head-lockedaudio to spatial audio, etc.), wherein the first audio output mode hasfewer computationally-controlled variables (e.g., volume of each soundsource, phase of each sound source, number of sound sources, activationsequence of available sound sources, etc.) than the second audio outputmode.

Changing an audio output mode from a first audio output mode to a secondaudio output mode that has more computationally controlled variablesthat the first audio output mode, in accordance with a determinationthat the biometric data corresponding to the first user meets the firstcriteria, provides improved audio feedback to the user (e.g., improvedaudio feedback that the computer system has transitioned from the firstlevel of immersion to the second level of immersion, improved audiofeedback that the biometric data corresponding to the first user has metthe first criteria, etc.). Providing improved feedback enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

It should be understood that the particular order in which theoperations in FIG. 10 have been described is merely an example and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 8000, 9000, 11000, and 12000) are also applicable in ananalogous manner to method 10000 described above with respect to FIG. 10. For example, the gestures, gaze inputs, physical objects, userinterface objects, controls, movements, criteria, three-dimensionalenvironment, display generation component, surface, representation ofphysical object, virtual objects, and/or animations described above withreference to method 10000 optionally have one or more of thecharacteristics of the gestures, gaze inputs, physical objects, userinterface objects, controls, movements, criteria, three-dimensionalenvironment, display generation component, surface, representation ofphysical object, virtual objects, and/or animations described hereinwith reference to other methods described herein (e.g., methods 8000,9000, 11000, and 12000). For brevity, these details are not repeatedhere.

FIG. 11 is a flowchart of a method of aggregating the effects ofmultiple types of the sensory adjustment provided by a computer systemwhen displaying a view of an environment that includes a representationof a physical environment, in accordance with some embodiments.

In some embodiments, the method 11000 is performed at a computer system(e.g., computer system 101 in FIG. 1 ) including a display generationcomponent (e.g., display generation component 120 in FIGS. 1, 3, and 4 )(e.g., a heads-up display, a display, a touchscreen, a projector, etc.)and one or more cameras (e.g., a camera (e.g., color sensors, infraredsensors, and other depth-sensing cameras) that points downward at auser's hand or a camera that points forward from the user's head). Insome embodiments, the method 11000 is governed by instructions that arestored in a non-transitory computer-readable storage medium and that areexecuted by one or more processors of a computer system, such as the oneor more processors 202 of computer system 101 (e.g., control unit 110 inFIG. 1A). Some operations in method 11000 are, optionally, combinedand/or the order of some operations is, optionally, changed.

In some embodiments, the method 11000 is performed at a computer system(e.g., computer system 101 in FIG. 1 ) that is in communication with adisplay generation component (e.g., display generation component 120 inFIGS. 1, 3, and 4 , display generation component 7100, etc.) (e.g., aheads-up display, an HMD, a display, a touchscreen, a projector, etc.)and one or more input devices (e.g., cameras, controllers,touch-sensitive surfaces, joysticks, buttons, etc.). In someembodiments, the computer system is an integrated device with one ormore processors and memory enclosed in the same housing as the displaygeneration component and at least some of the one or more input devices.In some embodiments, the computer system includes a computing componentthat includes one or more processors and memory that is separate fromthe display generation component and/or the one or more input devices.In some embodiments, the display generation component and the one ormore input devices are integrated and enclosed in the same housing.

The computer system displays (11002) a first view (e.g., view 7340 inFIG. 7K, or another view, etc.) of a physical environment, wherein thefirst view of the physical environment includes a first representation(e.g., representations 7350′, 7348′, etc. in FIG. 7K) of a first portionof the physical environment (e.g., the first representation is a regularcolor or B/W camera view of the first portion of the physicalenvironment, a view of the physical environment through a pass-throughportion of the display generation component, etc.) (e.g., the firstrepresentation is a baseline representation that is displayed withoutone or more types of computer-generated sensory enhancement). Whiledisplaying the first view of the physical environment, the computersystem detects (11004) a first user input (e.g., selection of a firstuser interface control, activation of a first hardware button in a firstmanner, performance of a first predefined gesture input, utterance of afirst preset voice command, etc.) that corresponds to a request toactivate a first type of computer-generated sensory adjustment (e.g.,binocular, heat vision, microscope, etc.) of two or more types ofcomputer-generated sensory adjustments (e.g., binocular vision, heatvision, microscope vision, night vision, super hearing, etc.). Inresponse to detecting the first user input, the computer system displays(11006) a second view of the physical environment (e.g., second view7361 shown in FIG. 7L, or another view, etc.), wherein the second viewof the physical environment includes a second representation (e.g.,representations 7350″, 7348″, etc. in FIG. 7L) of the first portion ofthe physical environment, wherein the second representation of the firstportion of the physical environment has a first display property (e.g.,resolution, zoom level, magnification, color distribution, intensitydistribution, focus distance, etc.) that is adjusted relative to thefirst representation of the first portion of the physical environment inaccordance with the first type of computer-generated sensory adjustment(e.g., binocular vision, microscope vision, heat vision, night vision,etc.). In some embodiments, the representation of the first portion ofthe physical environment is changed relative to the baselinerepresentation in terms of the size, resolution, focus distance,magnification of the subject matter captured in the representation aswell as the distributions of colors and light intensities due to theenhancement and/suppression of different portions of a light and/orcolor spectrum by the applied sensory adjustment. In the example shownin FIG. 7K-7L, the representations 7350″ and 7348″ are enlarged and/ormoved closer to the viewpoint of the view 7361, as compared to therepresentations 7350′ and 7348′ in the view 7340. While displaying thesecond view of the physical environment (e.g., the view 7361 in FIG. 7L,or another view, etc.), the computer system detects (11008) a seconduser input (e.g., selection of a second user interface control,activation of the first hardware button in a second manner, activationof a second hardware button in the first manner, performance of a secondpredefined gesture input, utterance of a first preset voice command,etc.) that corresponds to a request to activate a second type ofcomputer-generated sensory adjustment of the two or more types ofcomputer-generated sensory adjustments, wherein the second type ofcomputer-generated sensory adjustment is different from the first typeof computer-generated sensory adjustment. In response to detecting thesecond user input, the computer system displays (11010) a third view(e.g., view 7364, or another view, etc.) of the physical environment,wherein the third view of the physical environment incudes a thirdrepresentation (e.g., representations 7350′ and 7348′, etc.) of thefirst portion of the physical environment, wherein the thirdrepresentation of the first portion of the physical environment has thefirst display property (e.g., resolution, zoom level, magnification,color distribution, intensity distribution, focus distance, etc.) thatis adjusted relative to the first representation of the first portion ofthe physical environment in accordance with the first type ofcomputer-generated sensory adjustment (e.g., binocular vision,microscope vision, night vision, heat vision, etc.), and a seconddisplay property (e.g., resolution, zoom level, magnification, colordistribution, intensity distribution, focus distance, etc.) that isadjusted relative to the second representation of the physicalenvironment (e.g., the second display property have the same values inthe first representation and the second representation in somecombinations of the first type and second type of sensory enhancements;the second display property have different values in the firstrepresentation and the second representation in some combinations of thefirst type and second type of sensory enhancements) in accordance withthe second type of computer-generated sensory adjustment (e.g.,binocular vision, microscope vision, night vision, heat vision, colorfilter, etc.). In the example shown in FIG. 7K-7M, the representations7350″ and 7348″ are enlarged and/or moved closer to the viewpoint of theview 7361, as compared to the representations 7350′ and 7348′ in theview 7340; and the representations 7350′″ and 7348′ are changed in colorand intensity, as compared to the representations 7350″ and 7348″.

In some embodiments, while displaying the third view (e.g., view 7364 inFIG. 7M, or another view, etc.) of the physical environment, thecomputer system detects a third user input that corresponds to a requestto activate a third type of computer-generated sensory adjustment (e.g.,adjustment function corresponding to affordance 7358 in FIG. 7M, anotheradjustment function that has not yet been activated, etc.) (e.g.,binocular vision, microscope vision, night vision, heat vision, colorfilter, etc.) of the two or more types of computer-generated sensoryadjustments, wherein the third type of computer-generated sensoryadjustment is different from the first type of computer-generatedsensory adjustment and the second type of computer-generated sensoryadjustment, and in response to detecting the third user input, thecomputer system displays a fourth view of the physical environment,wherein the fourth view of the physical environment incudes a fourthrepresentation of the first portion of the physical environment, whereinthe fourth representation of the first portion of the physicalenvironment has the first display property (e.g., resolution, zoomlevel, magnification, color distribution, intensity distribution, focusdistance, etc.) that is adjusted relative to the first representation ofthe first portion of the physical environment in accordance with thefirst type of computer-generated sensory adjustment (e.g., binocularvision, microscope vision, night vision, heat vision, color filter,etc.), the second display property (e.g., resolution, zoom level,magnification, color distribution, intensity distribution, focusdistance, etc.) that is adjusted relative to the second representationof the physical environment in accordance with the second type ofcomputer-generated sensory adjustment (e.g., binocular vision,microscope vision, night vision, heat vision, color filter, etc.), and athird display property (e.g., resolution, zoom level, magnification,color distribution, intensity distribution, focus distance, etc.) thatis adjusted relative to the third representation of the physicalenvironment in accordance with the third type of computer-generatedsensory adjustment (e.g., binocular vision, microscope vision, nightvision, heat vision, color filter, etc.).

Displaying a fourth view of the physical environment including a fourthrepresentation of the first portion of the physical environment with thefirst display property that is adjusted relative to the firstrepresentation of the first portion of the physical environment inaccordance with the first type of computer-generated sensory adjustment,in response to detecting the third user input that corresponds to arequest to activate a third type of computer-generated sensoryadjustment of the two or more types of computer-generated sensoryadjustments, provides additional control options without cluttering theUI with additional displayed controls (e.g., additional displayedcontrols for selecting and/or activating the first, second, or thirdtype of computer-generated sensory adjustment). Providing additionalcontrol options without cluttering the UI with additional displayedcontrols enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated telescope vision (e.g., illustrated inFIGS. 7K-7L) (e.g., binocular vision, monocular vision, telescopevision, etc.) (e.g., reducing focus distance of objects such that theyappear closer to the user) for viewing distant physical objects, and thesecond type of computer-generated sensory adjustment includes simulatedmicroscope vision for magnifying nearby physical objects. In someembodiments, displaying the first representation of the physicalenvironment includes displaying a representation of a distant physicalobject at a first virtual position (e.g., with corresponding size anddisplay resolution for that virtual position in the three-dimensionalenvironment displayed via the first display generation component) thatcorresponds to the location of the distant physical object in thephysical environment. For example, the first representation of thedistant physical object also appears far away in the firstrepresentation of the physical environment, as the distant physicalobject appears in the physical environment. Displaying the secondrepresentation of the physical environment includes displaying arepresentation of the distant physical object at a second virtualposition that is closer to the viewpoint or virtual position of the userthan the first virtual position (e.g., with corresponding size anddisplay resolution for the second virtual position in thethree-dimensional environment displayed via the first display generationcomponent). For example, the second representation of the distantphysical object appears less far away in the second representation ofthe physical environment, and occupies a larger portion of the user'sfield of view of the second representation of the physical environment.Displaying the third representation of the physical environment includesdisplaying a representation of the distant physical object at a thirdvirtual position that is optionally even closer to the viewpoint orvirtual position of the user than the second virtual position and with apositive magnification (e.g., 100 times, 20 times, etc.) relative to thesize of the second representation of the distant physical object (e.g.,the distant physical object appear to be magnified at the second orthird virtual position). In an example usage scenario, the displaygeneration component first displays a camera view of a tree a firstdistance (e.g., 30 meters, 50 meters, etc.) away; then with telescopeview activated, the display generation component displays a telescopeview of the tree at a virtual position that is a second distance (e.g.,5 meters, 10 meters, etc.) away from the viewpoint corresponding to thecurrently displayed representation of the physical environment; and withtelescope view and microscope view both activated, the displaygeneration component displays a magnified view of at least a portion ofthe tree (e.g., 30× magnification, 100× magnification, etc.) at thecurrent virtual position (e.g., 5 meters, 10 meters, etc. away). In someembodiments, the camera view, the telescope view, and the microscopeview of the same portion of a physical object are, optionally, capturedby different cameras and/or sensors, or, optionally, enhanced withcomputational techniques.

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated telescope vision for viewingdistant physical objects, in response to detecting the first user input,and displaying a third view of the physical environment that includes athird representation of the first portion of the physical environment,wherein the third representation of the first portion of the physicalenvironment has the first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated telescope vision for viewingdistant physical objects, and a second display property that is adjustedrelative to the second representation of the physical environment inaccordance with simulated microscope vision for magnifying nearbyphysical objects, in response to detecting the second user input,provides additional control options without cluttering the UI withadditional displayed controls (e.g., additional displayed controls forselecting or switching between simulated telescope vision for viewingdistant objects, and simulated microscope vision for magnifying nearbyphysical objects). Providing additional control options withoutcluttering the UI with additional displayed controls enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated telescope vision (e.g., illustrated in7K-7L) (e.g., reducing focus distance of objects such that they appearcloser to the user) for viewing distant physical objects, and the secondtype of computer-generated sensory adjustment includes simulated nightvision (e.g., high sensitivity in low light conditions, brightness ofobjects are visually enhanced, small variations in brightness aremagnified, etc.) for viewing physical objects under low lightconditions. In some embodiments, displaying the first representation ofthe physical environment includes displaying a representation of adistant physical object at a first virtual position (e.g., withcorresponding size and display resolution for that virtual position inthe three-dimensional environment displayed via the first displaygeneration component) that corresponds to the location of the distantphysical object in the physical environment under low light conditions.For example, the first representation of the distant physical objectalso appears far away in the first representation of the physicalenvironment, as the distant physical object appears in the physicalenvironment, and the first representation of the physical environmentappears dark and objects are not clearly discernable due to the lowlight condition of the physical environment. Displaying the secondrepresentation of the physical environment includes displaying arepresentation of the distant physical object at a second virtualposition that is closer to the viewpoint or virtual position of the userthan the first virtual position (e.g., with corresponding size anddisplay resolution for the second virtual position in thethree-dimensional environment displayed via the first display generationcomponent), but still under low light conditions. For example, thesecond representation of the distant physical object appears less faraway in the second representation of the physical environment, andoccupies a larger portion of the user's field of view of the secondrepresentation of the physical environment, but the secondrepresentation of the physical environment still appears dark andobjects are not clearly discernable due to the low light condition ofthe physical environment. Displaying the third representation of thephysical environment includes displaying a representation of the distantphysical object at the second virtual position with enhanced brightnessand/or contrast (e.g., enhanced with images from low light cameras, orenhanced digitally by combining multiple photos and/or using machinelearning, etc.). In an example usage scenario, the display generationcomponent first displays a camera view of a tree a first displace (e.g.,30 meters, 50 meters, etc.) away during nighttime; then with telescopeview activated, the display generation component displays a telescopeview of the tree at a virtual position that is a second distance (e.g.,5 meters, 10 meters, etc.) away from the viewpoint corresponding to thecurrently displayed representation of the physical environment but thewhole scene is still dark due to the low light condition of the night;and with telescope view and night vision both activated, the displaygeneration component displays a brightened and high contrast image ofthe tree at the current virtual position (e.g., 5 meters, 10 meters,etc. away). In some embodiments, the camera view, the telescope view,and the night vision view of the same portion of a physical object are,optionally, captured by different cameras and/or sensors, or,optionally, enhanced with computational techniques.

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated telescope vision for viewingdistant physical objects, in response to detecting the first user input,and displaying a third view of the physical environment that includes athird representation of the first portion of the physical environment,wherein the third representation of the first portion of the physicalenvironment has the first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated telescope vision for viewingdistant physical objects, and a second display property that is adjustedrelative to the second representation of the physical environment inaccordance with simulated night vision for viewing physical objectsunder low light conditions, provides additional control options withoutcluttering the UI with additional displayed controls (e.g., additionaldisplayed controls for selecting or switching between simulatedtelescope vision for viewing distant objects, and simulated night visionfor viewing physical objects under low light conditions). Providingadditional control options without cluttering the UI with additionaldisplayed controls enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated telescope vision (e.g., illustrated in7K-7L) (e.g., reducing focus distance of objects such that they appearcloser to the user) for viewing distant physical objects, and the secondtype of computer-generated sensory adjustment includes simulated heatvision (e.g., illustrated in 7L-7M) (e.g., high sensitivity totemperature variations, presenting color and/or intensity variations inaccordance with temperature and/or thermal radiation variations, etc.)for viewing physical objects with different thermal radiation profiles.In some embodiments, displaying the first representation of the physicalenvironment includes displaying a representation of a distant physicalobject at a first virtual position (e.g., with corresponding size anddisplay resolution for that virtual position in the three-dimensionalenvironment displayed via the first display generation component) thatcorresponds to the location of the distant physical object in thephysical environment. For example, the first representation of thedistant physical object also appears far away in the firstrepresentation of the physical environment, as the distant physicalobject appears in the physical environment. Displaying the secondrepresentation of the physical environment includes displaying arepresentation of the distant physical object at a second virtualposition that is closer to the viewpoint or virtual position of the userthan the first virtual position (e.g., with corresponding size anddisplay resolution for the second virtual position in thethree-dimensional environment displayed via the first display generationcomponent). For example, the second representation of the distantphysical object appears less far away in the second representation ofthe physical environment, and occupies a larger portion of the user'sfield of view of the second representation of the physical environment.Displaying the third representation of the physical environment includesdisplaying a representation of the distant physical object at the secondvirtual position with its thermal radiation profile or temperature map.In an example usage scenario, the display generation component firstdisplays a camera view of a tree a first distance (e.g., 30 meters, 50meters, etc.) away; then with telescope view activated, the displaygeneration component displays a telescope view of the tree at a virtualposition that is a second distance (e.g., 5 meters, 10 meters, etc.)away from the viewpoint corresponding to the currently displayedrepresentation of the physical environment; and with telescope view andheat vision both activated, the display generation component displays aheat map of the tree at the current virtual position (e.g., 5 meters, 10meters, etc. away) showing a bright profile of a squirrel hidden amongthe tree leaves. In some embodiments, the camera view, the telescopeview, and the heat vision view of the same portion of a physical objectare, optionally, captured by different cameras and/or sensors, or,optionally, enhanced with computational techniques.

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated telescope vision for viewingdistant physical objects, in response to detecting the first user input,and displaying a third view of the physical environment that includes athird representation of the first portion of the physical environment,wherein the third representation of the first portion of the physicalenvironment has the first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated telescope vision for viewingdistant physical objects, and a second display property that is adjustedrelative to the second representation of the physical environment inaccordance with simulated heat vision for viewing physical objects withdifferent thermal radiation profiles, provides additional controloptions without cluttering the UI with additional displayed controls(e.g., additional displayed controls for selecting or switching betweensimulated telescope vision for viewing distant objects, and simulatedheat vision for viewing physical objects with different thermalradiation profiles). Providing additional control options withoutcluttering the UI with additional displayed controls enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated telescope vision (e.g., illustrated in7K-7L) (e.g., reducing focus distance of objects such that they appearcloser to the user) for viewing distant physical objects, and the secondtype of computer-generated sensory adjustment includes modifying a viewof physical objects with a filter (e.g., color filter, light frequencyfilter, intensity filter, a motion filter, etc.). In some embodiments,displaying the first representation of the physical environment includesdisplaying a representation of a distant physical object at a firstvirtual position (e.g., with corresponding size and display resolutionfor that virtual position in the three-dimensional environment displayedvia the first display generation component) that corresponds to thelocation of the distant physical object in the physical environment. Forexample, the first representation of the distant physical object alsoappears far away in the first representation of the physicalenvironment, as the distant physical object appears in the physicalenvironment. Displaying the second representation of the physicalenvironment includes displaying a representation of the distant physicalobject at a second virtual position that is closer to the viewpoint orvirtual position of the user than the first virtual position (e.g., withcorresponding size and display resolution for the second virtualposition in the three-dimensional environment displayed via the firstdisplay generation component). For example, the second representation ofthe distant physical object appears less far away in the secondrepresentation of the physical environment, and occupies a largerportion of the user's field of view of the second representation of thephysical environment. Displaying the third representation of thephysical environment includes displaying a representation of the distantphysical object at the second virtual position with some of the colors,and/or intensities, etc. filtered out. In some embodiments, when amotion filter is applied, parts of the second representation of thephysical environment that do not have motion are filtered out,highlighting parts with motion (e.g., movement of leaves, animals,people, etc.). In an example usage scenario, the display generationcomponent first displays a camera view of a tree a first distance (e.g.,30 meters, 50 meters, etc.); then with telescope view activated, thedisplay generation component displays a telescope view of the tree at avirtual position that is a second distance (e.g., 5 meters, 10 meters,etc.) away from the viewpoint corresponding to the currently displayedrepresentation of the physical environment; and with telescope view andcolor/intensity/motion filters both activated, the display generationcomponent displays a filtered image of the tree at the current virtualposition (e.g., 5 meters, 10 meters, etc. away) showing a bright orangecolored hat and safety vest on a faint de-saturated image of the tree(color filter applied), or a filtered image of the tree at the currentvirtual position (e.g., 5 meters, 10 meters, etc. away) showing visualhighlighting of a camouflaged insect moving on a faint de-saturatedimage of the tree. In some embodiments, the camera view, and thetelescope view of the same portion of a physical object are, optionally,captured by different cameras and/or sensors, or, optionally, enhancedwith computational techniques.

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated telescope vision for viewingdistant physical objects, in response to detecting the first user input,and displaying a third view of the physical environment that includes athird representation of the first portion of the physical environment,wherein the third representation of the first portion of the physicalenvironment has the first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated telescope vision for viewingdistant physical objects, and a second display property that is adjustedrelative to the second representation of the physical environment inaccordance with a filter that modifies a view of physical objects,provides additional control options without cluttering the UI withadditional displayed controls (e.g., additional displayed controls forselecting or switching between simulated telescope vision for viewingdistant objects, and the filter that modifies a view of physicalobjects). Providing additional control options without cluttering the UIwith additional displayed controls enhances the operability of thedevice, which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated telescope vision (e.g., illustrated in7K-7L) (e.g., reducing focus distance of objects such that they appearcloser to the user) for viewing distant physical objects, and the secondtype of computer-generated sensory adjustment includes selective audioenhancement (e.g., enhancing volume, selectively enhancing/suppressingcertain sound frequencies, etc.) for sounds corresponding to a subset ofphysical objects (e.g., a selected subset of all sound producingphysical objects, physical objects that are in the center of the currentfield of view, etc.) in a physical environment. In some embodiments,displaying the first representation of the physical environment includesdisplaying a representation of a distant physical object at a firstvirtual position (e.g., with corresponding size and display resolutionfor that virtual position in the three-dimensional environment displayedvia the first display generation component) that corresponds to thelocation of the distant physical object in the physical environment. Forexample, the first representation of the distant physical object alsoappears far away in the first representation of the physicalenvironment, as the distant physical object appears in the physicalenvironment. Displaying the second representation of the physicalenvironment includes displaying a representation of the distant physicalobject at a second virtual position that is closer to the viewpoint orvirtual position of the user than the first virtual position (e.g., withcorresponding size and display resolution for the second virtualposition in the three-dimensional environment displayed via the firstdisplay generation component). For example, the second representation ofthe distant physical object appears less far away in the secondrepresentation of the physical environment, and occupies a largerportion of the user's field of view of the second representation of thephysical environment. Displaying the third representation of thephysical environment includes displaying a representation of the distantphysical object at the second virtual position with visualidentification of a localized sound source in the physical environmenton or in the vicinity of the representation of the distant physicalobject, wherein the enhanced audio output corresponding to the soundfrom the localized sound source is output with the display of the thirdrepresentation of the physical environment. In an example usagescenario, the display generation component first displays a camera viewof a tree a first distance (e.g., 30 meters, 50 meters, etc.) awayduring nighttime; then with telescope view activated, the displaygeneration component displays a telescope view of the tree at a virtualposition that is a second distance (e.g., 5 meters, 10 meters, etc.)away from the viewpoint corresponding to the currently displayedrepresentation of the physical environment; and with telescope view andenhanced hearing both activated, the display generation componentdisplays a circle overlaid on the image of the tree at the currentvirtual position (e.g., 5 meters away, 3 meters away, etc.) indicating aposition of a bird singing in the tree. The localized chirping soundfrom the bird is played back along with the view of the three at thesecond distance (e.g., 5 meters, 10 meters, etc.) away from theviewpoint, optionally, with a spatial audio output mode. In someembodiments, the camera view, the telescope view, and the localizedsounds of the same portion of a physical object are, optionally,captured by different cameras and/or sensors, or, optionally, enhancedwith computational techniques.

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated telescope vision for viewingdistant physical objects, in response to detecting the first user input,and displaying a third view of the physical environment that includes athird representation of the first portion of the physical environment,wherein the third representation of the first portion of the physicalenvironment has the first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated telescope vision for viewingdistant physical objects, and a second display property that is adjustedrelative to the second representation of the physical environment inaccordance with selective audio adjustment for sounds corresponding to asubset of physical objects in a physical environment, providesadditional control options without cluttering the UI with additionaldisplayed controls (e.g., additional displayed controls for selecting orswitching between simulated telescope vision for viewing distantobjects, and selective audio adjustment for sounds corresponding to asubset of physical objects in a physical environment). Providingadditional control options without cluttering the UI with additionaldisplayed controls enhances the operability of the device, which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, concurrently with displaying the thirdrepresentation of the physical environment (e.g., the representationshown in FIG. 7M, or another representation, etc.), the computer systemoutputs sounds that correspond to a first portion of the physicalenvironment (e.g., portions 7366″ and 7368″ in FIG. 7M, or anotherportion, etc.) visible in the third representation of the physicalenvironment, wherein the sounds are selectively enhanced (e.g.,increased in volume, with modifications to the amplitudes of someselected frequencies, etc.) relative to sounds from sources outside ofthe first portion of the physical environment. In an example scenario,two trees are visible in the first representation of the physicalenvironment along with audio output of sounds captured from the entirephysical environment; when the first tree is viewed with the telescopeview and enhanced hearing activated, the sound of bird chirping in thefirst tree are enhanced (e.g., made louder) relative to the sound ofsquirrels rustling in the second tree, and played with spatial audio tohave a virtual position corresponding to the virtual position of thefirst tree. Outputting sounds that correspond to a first portion of thephysical environment visible in the second representation of thephysical environment, wherein the sounds are selectively enhancedrelative to sounds from sources outside of the first portion of thephysical environment, concurrently with displaying the thirdrepresentation of the physical environment, provides improved audiofeedback to the user (e.g., improved audio feedback regarding the firstportion of the physical environment). Providing improved feedbackenhances the operability of the device, which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, concurrently with displaying the thirdrepresentation of the physical environment, the computer system displaystextual output corresponding to speech coming from a first portion ofthe physical environment (e.g., portions 7366″ and 7368″ in FIG. 7M, oranother portion, etc.) visible in both the second representation andthird representation of the physical environment, wherein the speech isselectively enhanced relative to sounds from sources outside of thefirst portion of the physical environment. In an example scenario, atree and a house are visible in the first representation of the physicalenvironment along with audio output of sounds captured from the entirephysical environment; when the house is viewed with the telescope viewand enhanced hearing activated, the sound of speech from the house areenhanced (e.g., made louder and more clear, etc.) relative to the soundof bird chirping in the tree, and textual output, such as subtitles,transcriptions, translations, are displayed. In some embodiments, thespeech sounds are replaced with corresponding audio translations.Displaying textual output corresponding to speech coming from a firstportion of the physical environment visible in both the secondrepresentation and third representation of the physical environment,wherein the speech is selectively enhanced relative to sounds fromsources outside of the first portion of the physical environment,concurrently with displaying the third representation of the physicalenvironment, provides improved visual feedback to the user (e.g.,improved visual feedback regarding speech coming from the first portionof the physical environment). Providing improved feedback enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated microscope vision for magnifying nearbyphysical objects, and the second type of computer-generated sensoryadjustment includes simulated heat vision (e.g., illustrated in FIGS.7L-7M) (e.g., high sensitivity to temperature variations, presentingcolor and/or intensity variations in accordance with temperature and/orthermal radiation variations, etc.) for viewing physical objects withdifferent thermal radiation profiles. In an example usage scenario, thedisplay generation component first displays a camera view of a microchipin a mobile phone; then with microscope view activated, the displaygeneration component displays a magnified view of the microchip; andwith microscope view and heat vision both activated, the displaygeneration component displays a heat map of the microchip at the currentmagnification level, showing high temperature areas relative to lowtemperature areas on the microchip. In some embodiments, the cameraview, the microscope view, and the heat vision view of the same portionof a physical object are, optionally, captured by different camerasand/or sensors, or, optionally, enhanced with computational techniques.Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated microscope vision formagnifying nearby physical objects, in response to detecting the firstuser input, and displaying a third view of the physical environment thatincludes a third representation of the first portion of the physicalenvironment, wherein the third representation of the first portion ofthe physical environment has the first display property that is adjustedrelative to the first representation of the first portion of thephysical environment in accordance with simulated microscope vision formagnifying nearby physical objects, and a second display property thatis adjusted relative to the second representation of the physicalenvironment in accordance with simulated heat vision for viewingphysical objects with different thermal radiation profiles, providesadditional control options without cluttering the UI with additionaldisplayed controls (e.g., additional displayed controls for selecting orswitching between simulated microscope vision for magnifying nearbyphysical objects, and simulated heat vision for viewing physical objectswith different thermal radiation profiles). Providing additional controloptions without cluttering the UI with additional displayed controlsenhances the operability of the device, which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated night vision (e.g., high sensitivity inlow light conditions, brightness of objects are visually enhanced, smallvariations in brightness are magnified, etc.) for viewing physicalobjects under low light conditions, and the second type ofcomputer-generated sensory adjustment includes simulated telescopevision (e.g., illustrated in FIG. 7K-7L) (e.g., reducing focus distanceof objects such that they appear closer to the user) for viewing distantphysical objects. In an example usage scenario, the display generationcomponent first displays a camera view of a tree a first distance (e.g.,30 meters, 60 meters, etc.) away at night, and the details of the treeare barely visible due to the low light conditions; then with nightvision activated, the display generation component displays a brightenedand high contrast image of the camera view showing the tree at the firstdistance (e.g., 30 meters, 60 meters, etc.) away; and with night visionand telescope view both activated, the display generation componentdisplays the brightened and high contrast image of the tree at a virtualposition 5 meters away.

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated night vision for viewingphysical objects under low light conditions, in response to detectingthe first user input, and displaying a third view of the physicalenvironment that includes a third representation of the first portion ofthe physical environment, wherein the third representation of the firstportion of the physical environment has the first display property thatis adjusted relative to the first representation of the first portion ofthe physical environment in accordance with simulated night vision forviewing physical objects under low light conditions, and a seconddisplay property that is adjusted relative to the second representationof the physical environment in accordance with simulated telescopevision for viewing distant physical objects, provides additional controloptions without cluttering the UI with additional displayed controls(e.g., additional displayed controls for selecting or switching betweensimulated night vision for viewing physical objects under low lightconditions, and simulated telescope vision for viewing distant physicalobjects). Providing additional control options without cluttering the UIwith additional displayed controls enhances the operability of thedevice, which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated night vision (e.g., high sensitivity inlow light conditions, brightness of objects are visually enhanced, smallvariations in brightness are magnified, etc.) for viewing physicalobjects under low light conditions, and the second type ofcomputer-generated sensory adjustment includes simulated microscopevision for magnifying nearby physical objects. In an example usagescenario, the display generation component first displays a camera viewof a table top in a dark room. The details of the room are barelyvisible due to the low light conditions; then with night visionactivated, the display generation component displays a brightened andhigh contrast image of the room showing some coins on top of the table;and with night vision and microscope view both activated, the displaygeneration component displays the brightened and high contrast image ofthe coins magnified showing details of the coins. In some embodiments,the microscope images of the coins are optionally captured at adifferent time from that of the night vision images, and/or from thecamera view of the table top. The information from different types ofsensors are cameras are combined to generate the third representation ofthe physical environment. In some embodiments, information (e.g., sizeof the coins, characteristics of images on the coins, etc.) extractedfrom the first representation and/or second representation of thephysical environment are used as the basis to obtain (e.g., from onlinesources, image databases, etc.) additional information (e.g., types ofthe coins, year of the coins, materials of the coins, etc.) about thedetails of the physical environment to generate the third representationof the physical environment (e.g., showing more details of the coinsthat are not captured by the first representation and the secondrepresentation).

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated night vision for viewingphysical objects under low light conditions, in response to detectingthe first user input, and displaying a third view of the physicalenvironment that includes a third representation of the first portion ofthe physical environment, wherein the third representation of the firstportion of the physical environment has the first display property thatis adjusted relative to the first representation of the first portion ofthe physical environment in accordance with simulated night vision forviewing physical objects under low light conditions, and a seconddisplay property that is adjusted relative to the second representationof the physical environment in accordance with simulated microscopevision for magnifying nearby physical objects, provides additionalcontrol options without cluttering the UI with additional displayedcontrols (e.g., additional displayed controls for selecting or switchingbetween simulated night vision for viewing physical objects under lowlight conditions, and simulated microscope vision for magnifying nearbyphysical objects). Providing additional control options withoutcluttering the UI with additional displayed controls enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated night vision (e.g., high sensitivity inlow light conditions, brightness of objects are visually enhanced, smallvariations in brightness are magnified, etc.) for viewing physicalobjects under low light conditions, and the second type ofcomputer-generated sensory adjustment includes simulated heat vision(e.g., illustrated in FIGS. 7L-7M) (e.g., high sensitivity totemperature variations, presenting color and/or intensity variations inaccordance with temperature and/or thermal radiation variations, etc.)for viewing physical objects with different thermal radiation profiles.In an example usage scenario, the display generation component firstdisplays a camera view of a tree in the night. The details of the treeare barely visible due to the low light conditions; then with nightvision activated, the display generation component displays a brightenedand high contrast image of the tree showing two nests; and with nightvision and heat vision both activated, the display generation componentdisplays the brightened and high contrast image of the nests, with onenest having a different color and/or light intensity from the othernest, indicating that one nest has been inhabited by an animal recentlyand the other is not. In some embodiments, the camera view, the nightvision view, and the heat vision view of the same portion of a physicalobject are, optionally, captured by different cameras and/or sensors,or, optionally, enhanced with computational techniques.

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated night vision for viewingphysical objects under low light conditions, in response to detectingthe first user input, and displaying a third view of the physicalenvironment that includes a third representation of the first portion ofthe physical environment, wherein the third representation of the firstportion of the physical environment has the first display property thatis adjusted relative to the first representation of the first portion ofthe physical environment in accordance with simulated night vision forviewing physical objects under low light conditions, and a seconddisplay property that is adjusted relative to the second representationof the physical environment in accordance with simulated heat vision forviewing physical objects with different thermal radiation profiles,provides additional control options without cluttering the UI withadditional displayed controls (e.g., additional displayed controls forselecting or switching between simulated night vision for viewingphysical objects under low light conditions, and simulated heat visionfor viewing physical objects with different thermal radiation profiles).Providing additional control options without cluttering the UI withadditional displayed controls enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated night vision (e.g., high sensitivity inlow light conditions, brightness of objects are visually enhanced, smallvariations in brightness are magnified, etc.) for viewing physicalobjects under low light conditions, and the second type ofcomputer-generated sensory adjustment includes and the second type ofcomputer-generated sensory adjustment includes selective audioenhancement (e.g., enhancing volume, selectively enhancing/suppressingcertain sound frequencies, etc.) for sounds corresponding to a subset ofphysical objects (e.g., a selected subset of all sound producingphysical objects, physical objects that are in the center of the currentfield of view, etc.) in a physical environment. In some embodiments,displaying the first representation of the physical environment includesdisplaying a representation of a physical object in a low lightcondition. Displaying the second representation of the physicalenvironment includes displaying a brightened and high contrast image ofthe dark room with normal audio output of sound captured from the wholeroom. Displaying the third representation of the physical environmentincludes displaying the same brightened and high contrast image of thedark room with a localized sound source identified and visuallyhighlighted in the image, and with enhanced audio output correspondingto the sound from the localized sound source. In an example usagescenario, the display generation component first displays a camera viewof the dark room with no discernable sound; then with night visionactivated, the display generation component displays an enhancedbrightness and high contrast view of the dark room showing furniture andappliances in the room; and with night vision and enhanced hearing bothactivated, the display generation component displays a circle overlaidon the brightened and high contrast image of the dark room indicating aposition of a refrigerator which low frequency vibration sounds can beheard. The localized sound from the refrigerator is enhanced and playedback along with the night vision view of the room, optionally, with aspatial audio output mode and with enhancement of the frequencies in thevibrations of the refrigerator. In some embodiments, the camera view,the telescope view, and the localized sounds of the same portion of aphysical object are, optionally, captured by different cameras and/orsensors, or, optionally, enhanced with computational techniques. In someembodiments, a user input is detected selecting the source of sound(e.g., a tap on the refrigerator in the night vision view, another inputselecting another sound source, etc.) for which enhanced audio isrequested.

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated night vision for viewingphysical objects under low light conditions, in response to detectingthe first user input, and displaying a third view of the physicalenvironment that includes a third representation of the first portion ofthe physical environment, wherein the third representation of the firstportion of the physical environment has the first display property thatis adjusted relative to the first representation of the first portion ofthe physical environment in accordance with simulated night vision forviewing physical objects under low light conditions, and a seconddisplay property that is adjusted relative to the second representationof the physical environment in accordance with selective audioadjustment for sounds corresponding to a subset of physical objects in aphysical environment, provides additional control options withoutcluttering the UI with additional displayed controls (e.g., additionaldisplayed controls for selecting or switching between simulated nightvision for viewing physical objects under low light conditions, andselective audio adjustment for sounds corresponding to a subset ofphysical objects in a physical environment). Providing additionalcontrol options without cluttering the UI with additional displayedcontrols enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated heat vision (e.g., illustrated in FIGS.7L-7M) (e.g., high sensitivity to temperature variations, presentingcolor and/or intensity variations in accordance with temperature and/orthermal radiation variations, etc.) for viewing physical objects withdifferent thermal radiation profiles and the second type ofcomputer-generated sensory adjustment includes simulated telescopevision (e.g., illustrated in FIGS. 7K-7L) (e.g., reducing focus distanceof objects such that they appear closer to the user) for viewing distantphysical objects. In an example usage scenario, the display generationcomponent first displays a camera view of a forest; then with heatvision activated, one area of the forest in the heat vision view appearsto have a higher temperature than other areas of the forest in the heatvision view, and the area of the forest with the higher temperature is afirst distance (e.g., 50 meters, 100 meters, etc.) away in the heatvision view. When the heat vision view and the binocular view are bothactivated, the display generation component displays a telescope view ofthe area with the higher temperature at a virtual position that is asecond distance (e.g., 5 meters, 10 meters, etc.) away from theviewpoint corresponding to the currently displayed representation of thephysical environment. The heat map of the area with the highertemperature displayed at the virtual position at the second distance(e.g., 5 meters, 10 meters, etc.) away shows a representation of asmoldering dead tree trunk. In some embodiments, the camera view, thetelescope view, and the heat vision view of the same portion of aphysical object are, optionally, captured by different cameras and/orsensors, or, optionally, enhanced with computational techniques.

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated heat vision for viewingphysical objects with different thermal radiation profiles, in responseto detecting the first user input, and displaying a third view of thephysical environment that includes a third representation of the firstportion of the physical environment, wherein the third representation ofthe first portion of the physical environment has the first displayproperty that is adjusted relative to the first representation of thefirst portion of the physical environment in accordance with simulatedheat vision for viewing physical objects with different thermalradiation profiles, and a second display property that is adjustedrelative to the second representation of the physical environment inaccordance with simulated telescope vision for viewing distant physicalobjects, provides additional control options without cluttering the UIwith additional displayed controls (e.g., additional displayed controlsfor selecting or switching between simulated heat vision for viewingphysical objects with different thermal radiation profiles, andsimulated telescope vision for viewing distant physical objects).Providing additional control options without cluttering the UI withadditional displayed controls enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated heat vision (e.g., illustrated in FIGS.7L-7M) (e.g., high sensitivity to temperature variations, presentingcolor and/or intensity variations in accordance with temperature and/orthermal radiation variations, etc.) for viewing physical objects withdifferent thermal radiation profiles and the second type ofcomputer-generated sensory adjustment includes simulated microscopevision for magnifying nearby physical objects. In an example usagescenario, the display generation component first displays a camera viewof an internal structure of a mobile device; then with heat visionactivated, one area of the internal structure of the mobile deviceappears to have a higher temperature than other areas of the structurein the heat vision view, and the area of the structure with the highertemperature is shown with a modified color. When the heat vision viewand the microscope view are both activated, the display generationcomponent displays a magnified view of the area with the highertemperature. In some embodiments, the camera view, the telescope view,and the heat vision view of the same portion of a physical object are,optionally, captured by different cameras and/or sensors, or,optionally, enhanced with computational techniques.

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated heat vision for viewingphysical objects with different thermal radiation profiles, in responseto detecting the first user input, and displaying a third view of thephysical environment that includes a third representation of the firstportion of the physical environment, wherein the third representation ofthe first portion of the physical environment has the first displayproperty that is adjusted relative to the first representation of thefirst portion of the physical environment in accordance with simulatedheat vision for viewing physical objects with different thermalradiation profiles, and a second display property that is adjustedrelative to the second representation of the physical environment inaccordance with simulated microscope vision for magnifying nearbyphysical objects, provides additional control options without clutteringthe UI with additional displayed controls (e.g., additional displayedcontrols for selecting or switching between simulated heat vision forviewing physical objects with different thermal radiation profiles, andsimulated microscope vision for magnifying nearby physical objects).Providing additional control options without cluttering the UI withadditional displayed controls enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated heat vision (e.g., illustrated in FIGS.7L-7M) (e.g., high sensitivity to temperature variations, presentingcolor and/or intensity variations in accordance with temperature and/orthermal radiation variations, etc.) for viewing physical objects withdifferent thermal radiation profiles, and the second type ofcomputer-generated sensory adjustment includes simulated night vision(e.g., high sensitivity in low light conditions, brightness of objectsare visually enhanced, small variations in brightness are magnified,etc.) for viewing physical objects under low light conditions. In anexample usage scenario, the display generation component first displaysa camera view of a house in the night. The details of the house arebarely visible due to the low light conditions; then with heat visionactivated, the display generation component displays that one area ofthe heat vision view appear to have a higher temperature than otherparts of the heat vision view, but it is unclear what structural portionof the house hosts the high temperature area; and with night vision andheat vision both activated, the display generation component displaysthe brightened and high contrast image of the house, showing the hightemperature area inside a downspout running down the front side of thehouse. In some embodiments, the camera view, the night vision view, andthe heat vision view of the same portion of a physical object are,optionally, captured by different cameras and/or sensors, or,optionally, enhanced with computational techniques.

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated heat vision for viewingphysical objects with different thermal radiation profiles, in responseto detecting the first user input, and displaying a third view of thephysical environment that includes a third representation of the firstportion of the physical environment, wherein the third representation ofthe first portion of the physical environment has the first displayproperty that is adjusted relative to the first representation of thefirst portion of the physical environment in accordance with simulatedheat vision for viewing physical objects with different thermalradiation profiles, and a second display property that is adjustedrelative to the second representation of the physical environment inaccordance with simulated night vision for viewing physical objectsunder low light conditions, provides additional control options withoutcluttering the UI with additional displayed controls (e.g., additionaldisplayed controls for selecting or switching between simulated heatvision for viewing physical objects with different thermal radiationprofiles, and simulated night vision for viewing physical objects underlow light conditions). Providing additional control options withoutcluttering the UI with additional displayed controls enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

In some embodiments, the first type of computer-generated sensoryadjustment includes simulated heat vision (e.g., illustrated in FIGS.7L-7M) (e.g., high sensitivity to temperature variations, presentingcolor and/or intensity variations in accordance with temperature and/orthermal radiation variations, etc.) for viewing physical objects withdifferent thermal radiation profiles, and the second type ofcomputer-generated sensory adjustment includes selective audioenhancement (e.g., enhancing volume, selectively enhancing/suppressingcertain sound frequencies, etc.) for sounds corresponding to a subset ofphysical objects (e.g., a selected subset of all sound producingphysical objects, physical objects that are in the center of the currentfield of view, etc.) in a physical environment. In an example usagescenario, the display generation component first displays a camera viewof a house in the night. The details of the house are barely visible dueto the low light conditions; then with heat vision activated, thedisplay generation component displays that one area of the heat visionview appear to have a higher temperature than other parts of the heatvision view, but it is unclear what structural portion of the househosts the high temperature area; and with night vision and heat visionboth activated, the display generation component displays the brightenedand high contrast image of the house, showing the high temperature areainside a downspout running down the front side of the house; and withnight vision and enhanced hearing both activated, the display generationcomponent displays a circle overlaid on the high temperature areaindicating a position of sound source (e.g., a nest of snoring rodents).The localized sound from the highlighted sound source is enhanced andplayed back along with the heat vision view of the house, optionally,with a spatial audio output mode and with enhancement of the sound fromthe sound source. In some embodiments, the camera view, the heat visionview, and the localized sounds of the same portion of a physical objectare, optionally, captured by different cameras and/or sensors, or,optionally, enhanced with computational techniques. In some embodiments,a user input is detected selecting the source of sound (e.g., a tap onthe hot spot in the night vision view) for which enhanced audio isrequested.

Displaying a second view of the physical environment that includes asecond representation of the first portion of the physical environment,wherein the second representation of the first portion of the physicalenvironment has a first display property that is adjusted relative tothe first representation of the first portion of the physicalenvironment in accordance with simulated heat vision for viewingphysical objects with different thermal radiation profiles, in responseto detecting the first user input, and displaying a third view of thephysical environment that includes a third representation of the firstportion of the physical environment, wherein the third representation ofthe first portion of the physical environment has the first displayproperty that is adjusted relative to the first representation of thefirst portion of the physical environment in accordance with simulatedheat vision for viewing physical objects with different thermalradiation profiles, and a second display property that is adjustedrelative to the second representation of the physical environment inaccordance with selective audio adjustment for sounds corresponding to asubset of physical objects in a physical environment, providesadditional control options without cluttering the UI with additionaldisplayed controls (e.g., additional displayed controls for selecting orswitching between simulated heat vision for viewing physical objectswith different thermal radiation profiles, and selective audioadjustment for sounds corresponding to a subset of physical objects in aphysical environment). Providing additional control options withoutcluttering the UI with additional displayed controls enhances theoperability of the device, which, additionally, reduces power usage andimproves battery life of the device by enabling the user to use thedevice more quickly and efficiently.

It should be understood that the particular order in which theoperations in FIG. 11 have been described is merely an example and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 8000, 9000, 10000, and 12000) are also applicable in ananalogous manner to method 11000 described above with respect to FIG. 11. For example, the gestures, gaze inputs, physical objects, userinterface objects, controls, movements, criteria, three-dimensionalenvironment, display generation component, surface, representation ofphysical object, virtual objects, and/or animations described above withreference to method 11000 optionally have one or more of thecharacteristics of the gestures, gaze inputs, physical objects, userinterface objects, controls, movements, criteria, three-dimensionalenvironment, display generation component, surface, representation ofphysical object, virtual objects, and/or animations described hereinwith reference to other methods described herein (e.g., methods 8000,9000, 10000, and 12000). For brevity, these details are not repeatedhere.

FIG. 12 is a flowchart of a method of selectively displaying virtualcontent that corresponds to a respective type of exercise in a view of athree-dimensional environment in accordance with a determination thatthe portion of the physical environment in the view of thethree-dimensional environment corresponds to the respective type ofexercise, in accordance with some embodiments.

In some embodiments, the method 12000 is performed at a computer system(e.g., computer system 101 in FIG. 1 ) including a display generationcomponent (e.g., display generation component 120 in FIGS. 1, 3, and 4 )(e.g., a heads-up display, a display, a touchscreen, a projector, etc.)and one or more cameras (e.g., a camera (e.g., color sensors, infraredsensors, and other depth-sensing cameras) that points downward at auser's hand or a camera that points forward from the user's head). Insome embodiments, the method 12000 is governed by instructions that arestored in a non-transitory computer-readable storage medium and that areexecuted by one or more processors of a computer system, such as the oneor more processors 202 of computer system 101 (e.g., control unit 110 inFIG. 1A). Some operations in method 12000 are, optionally, combinedand/or the order of some operations is, optionally, changed.

In some embodiments, the method 12000 is performed at a computer system(e.g., computer system 101 in FIG. 1 ) that is in communication with adisplay generation component (e.g., display generation component 120 inFIGS. 1, 3, and 4 , display generation component 7100, etc.) (e.g., aheads-up display, an HMD, a display, a touchscreen, a projector, etc.)and one or more input devices (e.g., cameras, controllers,touch-sensitive surfaces, joysticks, buttons, etc.). In someembodiments, the computer system is an integrated device with one ormore processors and memory enclosed in the same housing as the displaygeneration component and at least some of the one or more input devices.In some embodiments, the computer system includes a computing componentthat includes one or more processors and memory that is separate fromthe display generation component and/or the one or more input devices.In some embodiments, the display generation component and the one ormore input devices are integrated and enclosed in the same housing.

The computer system displays (12002) a first view (e.g., view 7405 inFIG. 7N(B), another view, etc.) of a three-dimensional environment(e.g., scene 105 in FIG. 7N(A), another physical environment, etc.)(e.g., a reality view with no virtual elements or minimal virtualelements, a reality view with user interface objects for controllingbasic functions of the computer system (e.g., application icons forlaunching different computer-generated experiences, display settings,audio controls, etc.), an augmented reality view displayed with alow-level of immersion (e.g., displaying user interface objects (e.g.,an application launch pad, a welcome user interface, a settings userinterface) that are not part of a specific application experience (e.g.,a health app, a meditation app, a workout app, a game app, etc.), andthat on aggregate only occupy a small percentage (e.g., less than 10%,less than 20%, etc.) of the user's field of view or are displayed inconfined floating windows, etc.), etc.), wherein the first view of thethree-dimensional environment includes a first representation of a firstportion of a physical environment (e.g., the first representation is aregular camera view of the first portion of the physical environmentsurrounding the user that is in a physical spatial relationship with thefirst display generation component to view the three-dimensionalenvironment via the first display generation component, a view of thephysical environment through a pass-through portion of the first displaygeneration component, etc.). While displaying the first view of thethree-dimensional environment including the first representation of thefirst portion of the physical environment, the computer system detects(12004) movement of a first user (e.g., user 7002 in FIGS. 7N-7P,another user, etc.) from a first location (e.g., location of user 7002in FIG. 7N(A), another location, etc.) to a second location (e.g.,location of user 7002 in FIG. 7O(A), location of user 7002 in FIG.7P(A), another location, etc.) of the physical environment (e.g.,movement of the first user as a whole (e.g., walking, climbing, etc.)while wearing an HMD that serves as the first display generationcomponent, movement of the first user carrying a mobile device with adisplay or projector that serves as the first display generationcomponent (e.g., movement of the user's arm that causes movement of themobile device and display, movement of the user as a whole carrying themobile device with the display) from a first location to a secondlocation in the physical environment, etc.). In response to detecting(12006) the movement of the first user from the first location to thesecond location and in accordance with a determination that the movementto the second location meets first criteria, wherein the first criteriainclude a first requirement that the second location corresponds to alocation associated with a first type of exercise (e.g., the locationhas a first type of exercise equipment (e.g., rowing machines, stairs,treadmill, climbing wall, stationary bikes, weight training machines,punching bags, etc.), the location is a location designed for (e.g.,having appropriate floor surface, mat, pool, walls, structures, etc.) afirst type of exercise (e.g., swimming, rowing, meditation, yoga,lifting, kicking, walking, running, dancing, climbing, playing tennis,playing basketball, doing gymnastics, etc.), etc.) in order for thefirst criteria to be met, the computer system displays (12008) a secondview (e.g., view 7408 in FIG. 7O(B), or another view, etc.) of thethree-dimensional environment (e.g., an augmented reality view with morevirtual elements corresponding to a first specific computer-generatedexperience corresponding to the current location, an augmented realityview showing a preview or start of a first computer-generated experiencecorresponding to the current location, an augmented reality viewdisplayed with a higher-level of immersion (e.g., displaying userinterface objects that are part of a first specific applicationexperience (e.g., virtual hiking trails, virtual scenery, score boards,exercise statistics, controls of changing exercise parameters, etc.),that on aggregate occupy a substantial percentage (e.g., greater than60%, greater than 90%, etc.) of the user's field of view or aredisplayed in a three-dimensional virtual or augmented realityenvironment, etc.), etc.), wherein the second view of thethree-dimensional environment includes a first set of virtual contentcorresponding to the first type of exercise (e.g., virtual open water7406 in FIG. 7O(B), other virtual content, etc.) (e.g., hiking trailscenery for a treadmill exercise program, a lake scene for a rowingmachine exercise, an arena for kickboxing, a virtual cliff side forclimbing wall exercise, a virtual tennis court for a virtual tennisgame, and/or user interface controls, scores, statistics, etc. for thefirst type of exercise, etc.), wherein the first set of virtual contentreplaces at least a portion of a second representation of a secondportion of the physical environment (e.g., the location of the user 7002in FIG. 7O(A), another location, etc.) (e.g., the virtual contentcorresponding to the first type of exercise is displayed overlaying,blocking the view of, replacing display of, etc.) of the representationof the portion of the physical environment (e.g., the view of the actualequipment for the first type of exercise or the room designed for thefirst type of exercise, etc.) including the second location). In someembodiments, the above requirement is an only requirement for the firstcriteria to be met. In some embodiments, the above requirement is arequirement alternative to one or more other requirements in the firstcriteria that don't have to all be met in order for the first criteriato be met. In some embodiments, the above requirement is a requirementin addition to one or more other requirements in the first criteria thatall have to be met in order for the first criteria to be met. In someembodiments, as an alternative condition (or an additional condition),the user has to perform an action associated with the exercise, forexample starting a characteristic motion (e.g., starting to walk on atreadmill, step on an stair stepper, move legs back and forth on anelliptical, or start rowing on a rowing machine), or steppingonto/sitting down on a piece of exercise equipment), in order for thefirst criteria to be met. In the method 12000, in response to detectingthe movement of the first user from the first location to the secondlocation in accordance with a determination that the movement to thesecond location meets second criteria, different from the firstcriteria, wherein the second criteria include a second requirement thatthe second location corresponds to a location associated with a secondtype of exercise (e.g., the location has a second type of exerciseequipment (e.g., rowing machines, stairs, treadmill, climbing wall,weight training machines, punching bags, etc.), the location is alocation designed for (e.g., having appropriate floor surface, mat,pool, walls, structures, etc.) a second type of exercise (e.g.,swimming, rowing, meditation, yoga, lifting, kicking, walking, running,dancing, climbing, playing tennis, playing basketball, doing gymnastics,etc.), etc.) in order for the second criteria to be met, wherein thesecond type of exercise is different from the first type of exercise,the computer system displays (12010) a third view (e.g., view 7410 inFIG. 7P(B), or another view, etc.) of the three-dimensional environment(e.g., an augmented reality view with more virtual elementscorresponding to a second specific computer-generated experiencecorresponding to the current location, an augmented reality view showinga preview or start of a second computer-generated experiencecorresponding to the current location, an augmented reality viewdisplayed with a higher-level of immersion (e.g., displaying userinterface objects that are part of a second specific applicationexperience (e.g., virtual hiking trails, virtual scenery, score boards,exercise statistics, controls of changing exercise parameters, etc.),that on aggregate occupy a substantial percentage (e.g., greater than60%, greater than 90%, etc.) of the user's field of view or aredisplayed in a three-dimensional virtual or augmented realityenvironment, etc.), etc.), wherein the third view of thethree-dimensional environment includes a second set of virtual contentcorresponding to the second type of exercise (e.g., hiking trail sceneryfor a treadmill exercise program, a lake scene for a rowing machineexercise, an arena for kickboxing, a virtual cliff side for climbingwall exercise, a virtual tennis court for a virtual tennis game, and/oruser interface controls, scores, statistics, etc. for the second type ofexercise, etc.), wherein the second set of virtual content is differentfrom the first set of virtual content (e.g., a virtual hiking trail vs.a virtual lake scene; a virtual tennis court vs. a virtual boxing ring;a virtual meadow for medication vs. a virtual stage for dancing, etc.),and wherein the second set of virtual content (e.g., virtual hikingtrail 7412 in FIG. 7P(B), other virtual content, etc.) replaces at leasta portion of a third representation of a third portion of the physicalenvironment (e.g., the virtual content corresponding to the second typeof exercise is displayed overlaying, blocking the view of, replacingdisplay of, etc. of the representation of the portion of the physicalenvironment (e.g., the view of the actual equipment for the first typeof exercise or the room designed for the first type of exercise, etc.))that includes the second location (e.g., the location of the user 7002in FIG. 7P(A), another location, etc.). In some embodiments, the aboverequirement is an only requirement for the first criteria to be met. Insome embodiments, the above requirement is a requirement alternative toone or more other requirements in the first criteria that don't have toall be met in order for the first criteria to be met. In someembodiments, the above requirement is a requirement in addition to oneor more other requirements in the first criteria that all have to be metin order for the first criteria to be met. In some embodiments, as analternative condition (or an additional condition), the user has toperform an action associated with the exercise, for example starting acharacteristic motion (e.g., starting to walk on a treadmill, step on anstair stepper, move legs back and forth on an elliptical, or startrowing on a rowing machine), or stepping onto/sitting down on a piece ofexercise equipment), in order for the first criteria to be met. Thesefeatures are illustrated in FIGS. 7N-7P, where when the user 7002 movesfrom location to location, depending on the current location of the user7002, the computer system determines which type of exercise isassociated with the current location of the user 7002. If the currentlocation is associated with a first type of exercise (e.g., locationthat includes object 7404), the computer system displays virtual content7408 (FIG. 7O) that corresponds to the first type of exercise (e.g.,rowing, boating, etc.). If the current location is associated a secondtype of exercise (e.g., location that includes object 7402), thecomputer system displays virtual content 7410 (FIG. 7P) that correspondsto the second type of exercise (e.g., hiking, walking, etc.).

In some embodiments, the computer system determines that the secondlocation corresponds to a location associated with the first type ofexercise in accordance with detection of a first type of exerciseequipment (e.g., object 7404 in FIG. 7O, other equipment, etc.) at thesecond location (e.g., detecting an RFID signal corresponding to thefirst type of exercise equipment at the second location, detecting animage of the first type of exercise equipment at the second location ina camera feed capturing the second location, detecting that the secondlocation matches a registered location for the first type of exerciseequipment, etc.) (e.g., the first type of exercise equipment isdifferent from the second type of exercise equipment and does notcorrespond to the second type of exercise). The computer systemdetermines that the second location corresponds to a location associatedwith the second type of exercise in accordance with detection of asecond type of exercise equipment (e.g., object 7402 in FIG. 7P, otherequipment, etc.) at the second location (e.g., detecting an RFID signalcorresponding to the second type of exercise equipment at the secondlocation, detecting an image of the second type of exercise equipment atthe second location in a camera feed capturing the second location,detecting that the second location is a registered location for thesecond type of exercise equipment, etc.), wherein the second type ofexercise equipment is different from the first type of exerciseequipment and does not correspond to the first type of exercise. Forexample, when the first user walks to a location in front of atreadmill, the HMD displays a virtual hiking trail that blocks, replacesdisplay of, or overlays the representation of the treadmill within theuser's field of view provided via the first display generationcomponent; and when the first user walks to a location in front of arowing machine, the HMD displays a virtual lake scene that blocks,replaces display of, or overlays the representation of the rowingmachine within the user's field of view provided via the first displaygeneration component. In some embodiments, the virtual content that isprovided by the first display generation component is automatically(e.g., without user inputs that specifically selects the virtual contentor program using a user interface element or voice command) changedbetween reality view, different augmented reality views with differentvirtual scenes, and/or different virtual environments, etc., when thefirst user moves from location to location (e.g., from the entrance ofthe gym to in front of the treadmill, from in front of the treadmill toin front of the rowing machine, etc.).

Displaying a second view of the three-dimensional environment thatincludes a first set of virtual content corresponding to the first typeof exercise, in accordance with a determination that the movement to thesecond location meets first criteria requiring that the second locationcorresponds to a location associated with a first type of exercise,wherein the computer system determines that the second locationcorresponds to a location associated with the first type of exercise inaccordance with detection of a first type of exercise equipment at thesecond location, and displaying a third view of the three-dimensionalenvironment includes a second set of virtual content, different from thefirst set of virtual content, corresponding to the second type ofexercise, in accordance with a determination that the movement to thesecond location meets second criteria, different from the firstcriteria, requiring that the second location corresponds to a locationassociated with a second type of exercise, wherein the computer systemdetermines that the second location corresponds to a location associatedwith the second type of exercise in accordance with detection of asecond type of exercise equipment at the second location, displays theappropriate set of virtual content when a set of conditions has been metwithout requiring further user input (e.g., further user input to selectthe set of virtual content corresponding to the first or second type ofexercise). Performing an operation when a set of conditions has been metwithout requiring further user input enhances the operability of thedevice, which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, displaying the second view (e.g., view 7408 in FIG.7O, or view 7410 in FIG. 7P, another view, etc.) of thethree-dimensional environment in response to detecting the movement ofthe first user from the first location to the second location in thephysical environment includes gradually reducing the secondrepresentation of the second portion of the physical environment (e.g.,a portion of the scene 105 that includes the object 7404 in FIG. 7O, anda portion of the scene 105 that includes the object 7402 in FIG. 7P,etc.) (e.g., ceasing display of more and more portions of therepresentation of the second portion of the physical environment, fadingout the representation of the second portion of the physicalenvironment, etc.), and gradually increasing a prominence of virtualcontent corresponding to the first type of exercise (e.g., starting todisplay the virtual content, increasing visibility of the virtualcontent, increasing a proportion of the field of view of the useroccupied by the virtual content, increasing an opacity or brightness ofthe virtual content, etc.) in regions of the second view of thethree-dimensional environment in which the second representation of thesecond portion of the physical environment has been gradually reduced.In some embodiments, displaying the third view of the three-dimensionalenvironment in response to detecting the movement of the first user fromthe first location to the second location in the physical environmentincludes: gradually reducing the representation of the third portion ofthe physical environment (e.g., ceasing display of more and moreportions of the representation of the third portion of the physicalenvironment, fading out the representation of the third portion of thephysical environment, etc.), and gradually increasing virtual contentcorresponding to the second type of exercise (e.g., displaying,increasing visibility of the virtual content, etc.) in regions of thethird view of the three-dimensional environment in which therepresentation of the third portion of the physical environment has beengradually reduced. For example, in some embodiments, when the first useris standing in front of a treadmill and/or stepped onto the treadmill,the user's view of the physical environment (e.g., the hardware controlpanel of the treadmill, the wall in front of the user, the otherexercise machines in the same room, etc.) is gradually altered, withmore and more portions of the representation of the physical environmentfading away and/or replaced with virtual content corresponding to thetreadmill exercise (e.g., hiking trail scenery, virtual paved patharound a virtual lake, etc.). Eventually, when the user starts walkingon the treadmill, the entire field of view of the user is filled withthe virtual scenery of the mountain trail or lakeside path. In someembodiments, the virtual content is a virtual three-dimensionalenvironment, and the user can view different portions of the virtualthree-dimensional environment by turning his/her head around, or up anddown, while walking on the treadmill.

Gradually reducing the second representation of the second portion ofthe physical environment, and gradually increasing prominence of virtualcontent corresponding to the first type of exercise in regions of thesecond view of the three-dimensional environment in which the secondrepresentation of the second portion of the physical environment hasbeen gradually reduced, provides improved visual feedback to the user(e.g., improved visual feedback that the computer system has detectedthe movement of the first user from the first location to the secondlocation in the physical environment). Providing improved feedbackenhances the operability of the device, which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, the first criteria include a third requirement thatthe movement of the first user from the first location to the secondlocation is followed by a first predefined movement corresponding to thefirst type of exercise (e.g., sitting on the object 7404 in FIG. 7O,stepping on object 7402 in FIG. 7P) (e.g., starting a characteristicmotion (e.g., starting to walk on a treadmill, stepping on an stairstepper, moving legs back and forth on an elliptical, or starting rowingon a rowing machine, etc.), stepping onto/sitting down on a piece ofexercise equipment corresponding to the first type of exercise (e.g.,sitting down on a rowing machine, or weight training machine, etc.),getting into a ready posture corresponding to the first type of exercise(e.g., standing in a ready posture for hitting a virtual tennis ball,sitting down on the floor to start meditation or yoga, etc.), etc.) inorder for the first criteria to be met. In some embodiments, the secondcriteria include a fourth requirement that the movement of the firstuser from the first location to the second location is followed by asecond predefined movement corresponding to the second type of exercise,wherein the second predefined movement is different from the first typeof movement. In some embodiments, the second predefined movement is thesame as the first predefined movement. For example, the predefinedmovement for starting a virtual environment for kickboxing is optionallythe same as the predefined movement requirement for starting a virtualenvironment for boxing. For example, the predefined movement forstarting a virtual environment for ballet is optionally different fromthe predefined movement requirement for starting a virtual environmentfor modern dance. In some embodiments, the computer system does notstart to display the virtual content corresponding to the first type ofexercise until the first predefined movement corresponding to the firsttype of exercise is detected, even if the first user is at the secondlocation and the second location is a location corresponding to thefirst type of exercise. In some embodiments, the computer systemdisplays a visual prompt for the first user to provide the firstpredefined movement to trigger display of the virtual content associatedwith the first type of exercise when the first user is detected at thesecond location and the second location is a location associated withthe first type of exercise.

Displaying a second view of the three-dimensional environment thatincludes a first set of virtual content corresponding to the first typeof exercise, in accordance with a determination that the movement to thesecond location meets first criteria, wherein the first criteria includea third requirement that the movement of the first user from the firstlocation to the second location is followed by a first predefinedmovement corresponding to the first type of exercise, providesadditional control options without cluttering the UI with additionaldisplayed controls (e.g., additional displayed controls for displayingthe first set of virtual content corresponding to the first type ofexercise, additional displayed controls for forgoing display of thefirst set of virtual content corresponding to the first type ofexercise, etc.). Providing additional control options without clutteringthe UI with additional displayed controls enhances the operability ofthe device, which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, in response to detecting the movement of the firstuser from the first location to the second location and in accordancewith a determination that the movement to the second location meetsthird criteria, different from the first criteria and the secondcriteria, wherein the third criteria include a requirement that thesecond location corresponds to a location associated with a third typeof exercise different from the first type of exercise and the secondtype of exercise (e.g., the second location optionally is associatedwith both the first type of exercise and the third type of exercise),and that the movement of the first user from the first location to thesecond location is followed by a third predefined movement correspondingto the third type of exercise (e.g., starting a characteristic motion(e.g., starting to walk on a treadmill, stepping on an stair stepper,moving legs back and forth on an elliptical, or starting rowing on arowing machine, etc.), stepping onto/sitting down on a piece of exerciseequipment corresponding to the respective type of exercise (e.g.,sitting down on a rowing machine, or weight training machine, etc.),getting into a ready posture corresponding to the respective type ofexercise (e.g., standing in a ready posture for hitting a virtual tennisball, sitting down on the floor to start meditation or yoga, etc.),etc.) in order for the third criteria to be met, wherein the thirdpredefined movement is different from the first predefined movement, thecomputer system displays a fourth view of the three-dimensionalenvironment (e.g., an augmented reality view with more virtual elementscorresponding to a third specific computer-generated experiencecorresponding to the current location, an augmented reality view showinga preview or start of a third computer-generated experiencecorresponding to the current location, an augmented reality viewdisplayed with a higher-level of immersion (e.g., displaying userinterface objects that are part of a third specific applicationexperience (e.g., virtual hiking trails, virtual scenery, score boards,exercise statistics, controls of changing exercise parameters, etc.),that on aggregate occupy a substantial percentage (e.g., greater than60%, greater than 90%, etc.) of the user's field of view or aredisplayed in a three-dimensional virtual or augmented realityenvironment, etc.), etc.). The fourth view of the three-dimensionalenvironment includes a third set of virtual content corresponding to thethird type of exercise (e.g., hiking trail scenery for a treadmillexercise program, a lake scene for a rowing machine exercise, an arenafor kickboxing, a virtual cliff side for climbing wall exercise, avirtual tennis court for a virtual tennis game, and/or user interfacecontrols, scores, statistics, etc. for the third type of exercise,etc.). The third set of virtual content is different from the first setof virtual content and the second set of virtual content, and whereinthe third set of virtual content replaces at least a portion of thesecond representation of the second portion of the physical environment(e.g., the second location corresponds to both the first type ofexercise and the third type of exercise, and whether the first set ofvirtual content or the third set of virtual content is displayed dependson whether the first predefined movement or the third predefinedmovement is detected while the first user is at the second location).

Displaying a fourth view of the three-dimensional environment thatincludes a third set of virtual content, different from the first set ofvirtual content and the second set of virtual content, corresponding tothe third type of exercise, in accordance with a determination that themovement to the second location meets third criteria, different from thefirst criteria and the second criteria, requiring that the secondlocation corresponds to a location associated with a third type ofexercise different from the first type of exercise and the second typeof exercise, and that the movement of the first user from the firstlocation to the second location is followed by a third predefinedmovement, different from the first predefined movement, corresponding tothe third type of exercise, provides additional control options withoutcluttering the UI with additional displayed controls (e.g., additionaldisplayed controls for displaying the first set of virtual contentcorresponding to the first type of exercise, additional displayedcontrols for displaying the third set of virtual content correspondingto the third type of exercise, etc.). Providing additional controloptions without cluttering the UI with additional displayed controlsenhances the operability of the device, which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, the computer system gradually increases an amountof virtual content displayed in a field of view of the first user (e.g.,in the view shown in FIGS. 7O(B) and 7P(B), etc.) in accordance with atleast one of a progress or duration of a predefined movementcorresponding to a respective type of exercise (e.g., the first type ofexercise, the second type of exercise, the third type of exercise, etc.)associated with the second location. For example, in some embodiments,the view of the real world gradually fades away and/or cease to bedisplayed, and is gradually replaced by virtual content corresponding tothe respective type of exercise. In some embodiments, the computersystem gradually increases the amount of virtual content displayed inthe field of view of the first user until a respective virtualenvironment corresponding to the respective type of exercise is fullydisplayed via the first display generation component (e.g., the secondview of the three-dimensional environment includes a virtual environmentcorresponding to the first type of exercise, the third view of thethree-dimensional environment includes a virtual environmentcorresponding to the second type of exercise, etc.). For example, insome embodiments, when an open gym is a location that is associated withboth yoga and dance, after the first user arrives at the open gym, ifthe first user sits down with a Namaste pose, the computer systemdisplays a virtual ocean view with ocean sounds for the user to practiceyoga on a virtual beach; and if the first user stands with a dancerpose, the computer system displays a virtual stage with dance music forthe first user to practice a dance.

Gradually increasing an amount of virtual content displayed in a fieldof view of the first user in accordance with at least one of a progressor duration of a predefined movement corresponding to a respective typeof exercise associated with the second location, provides improvedvisual feedback to the user (e.g., improved visual feedback regardingthe progress or duration of the predefined movement corresponding to therespective type of exercise associated with the second location).Providing improved feedback enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, while displaying a respective view of thethree-dimensional environment (e.g., augmented reality view, virtualreality view, etc.) that corresponds to a respective type of exercise(e.g., the first type of exercise, the second type of exercise, thethird type of exercise, etc.) associated with the second location, thecomputer system detects movement of the first user that corresponds to arequest to end the respective type of exercise associated with thesecond location (e.g., detecting the first user stopping the respectivetype of exercise, standing up, getting off the equipment, taking off theHMD, and/or walking away from the second location, etc.). In response todetecting the movement of the first user that corresponds to a requestto end the respective type of exercise associated with the secondlocation, the computer system detects a fifth view of thethree-dimensional environment that includes a representation of at leasta fourth portion of the physical environment, wherein the representationof at least the fourth portion of the physical environment occupies aportion of the field of view of the first user in which a respective setof virtual content that corresponds to the respective type of exercisehad been displayed while the first user was at the second location. Forexample, when the movement of the first user that corresponds to therequest to end the current exercise is detected, the virtual scenecorresponding to the current exercise ceases to be displayed (e.g., fadeaway, or cease to be displayed immediately, etc.) revealing therepresentation of the physical environment again. In some embodiments,when the user 7002 moves away from the object 7404 and have not reachedthe object 7402 in the scene 105, neither view 7408 nor view 7410 inFIGS. 7O and 7P are displayed, and a representation of the physicalenvironment such as that shown in FIG. 7N is displayed.

Displaying a fifth view of the three-dimensional environment thatincludes a representation of at least a fourth portion of the physicalenvironment, wherein the representation of at least the fourth portionof the physical environment occupies a portion of the field of view ofthe first user in which a respective set of virtual content thatcorresponds to the respective type of exercise had been displayed whilethe first user was at the second location, in response to detecting themovement of the first user that corresponds to a request to end therespective type of exercise associated with the second location,displays the fifth view of the three-dimensional environment when a setof conditions has been met without requiring further user input (e.g.,further user input to display the fifth view of the three-dimensionalenvironment). Performing an operation when a set of conditions has beenmet without requiring further user input enhances the operability of thedevice, which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, the computer system displays status information(e.g., progress, duration, speed, force, height, pace, stride length,performance level, scores, number of repetitions completed, etc. duringthe current session, historic statistics, average statistics for thefirst user and/or across multiple users, status of others alsoperforming the same type of exercise, etc.) corresponding to the firsttype of exercise when the second view (e.g., view 7408 in FIG. 7O, view7410 in FIG. 7P, another view, etc.) of the three-dimensionalenvironment is displayed. In some embodiments, the status informationcorresponding to the first type of exercise is overlaid on a portion ofthe virtual scene corresponding to the first type of exercise. In someembodiments, the status information is displayed in response to arequest of the first user that is detected while the virtual scenecorresponding to the first type of exercise is displayed without thestatus information. In some embodiments, the second view of thethree-dimensional environment evolves throughout the performance of thefirst type of exercise by the first user. In some embodiments, thestatus information is continuously updated throughout the performance ofthe first type of exercise by the first user (e.g., overlaying thechanging second view of the three-dimensional environment). In someembodiments, the status information is displayed in response todetecting that values of one or more performance parameters have metpreset threshold values (e.g., a target speed or distance is achieved, athreshold score is reached, etc.).

Displaying status information corresponding to the first type ofexercise when the second view of the three-dimensional environment isdisplayed provides improved visual feedback to the user (e.g., improvedvisual feedback regarding the first type of exercise, improved visualfeedback that the movement of the user to the second position satisfiesthe first criteria, improved visual feedback that the computer system isdisplaying the second view of the three-dimensional environment, etc.).Providing improved feedback enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently. In some embodiments, the computer system displays healthinformation (e.g., real-time biometric data (e.g., heart rate, bloodpressure, breathing rate, body temperature, blood sugar level, etc.),weight, BMI, etc.) corresponding to the first user when the second view(e.g., view 7408 in FIG. 7O, view 7410 in FIG. 7P, another view, etc.)of the three-dimensional environment is displayed. In some embodiments,the health information corresponding to the first user is overlaid on aportion of the virtual scene corresponding to the first type ofexercise. In some embodiments, the health information is displayed inresponse to a request of the first user that is detected while thevirtual scene corresponding to the first type of exercise is displayedwithout the health information. In some embodiments, the second view ofthe three-dimensional environment evolves throughout the performance ofthe first type of exercise by the first user. In some embodiments, thehealth information is continuously updated throughout the performance ofthe first type of exercise by the first user (e.g., overlaying thechanging second view of the three-dimensional environment). In someembodiments, the health information is displayed in response todetecting that values of one or more health parameters have met presetthreshold values (e.g., a target heart rate is achieve, a thresholdblood pressure is reached, etc.).

Displaying health information corresponding to the first user when thesecond view of the three-dimensional environment is displayed providesimproved visual feedback to the user (e.g., improved visual feedbackrelated to the first type of exercise, improved visual feedback that themovement of the user to the second position satisfies the firstcriteria, improved visual feedback that the computer system isdisplaying the second view of the three-dimensional environment, etc.).Providing improved feedback enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, the computer system visually presents progressinformation (e.g., real-time scores, laps completed, laps remaining,duration, number of steps, distance traveled, poses completed, etc.) ofthe first type of exercise that is performed by the first user when thesecond view (e.g., view 7408 in FIG. 7O, view 7410 in FIG. 7P, anotherview, etc.) of the three-dimensional environment is displayed. In someembodiments, the progress information is visually represented by thevisual changes occurring in the virtual scene that is presented to thefirst user (e.g., virtual milestones on a virtual hiking trail, numberof virtual shooting targets that are shown in the down position, scoreboards as part of the virtual game arena, the stillness of the water ona virtual lake represents the level of deep mediation that is achieved,etc.). In some embodiments, the progress information corresponding tothe performance of the first type of exercise by the first user isoverlaid on a portion of the virtual scene corresponding to the firsttype of exercise. In some embodiments, the progress information isdisplayed in response to a request of the first user that is detectedwhile the virtual scene corresponding to the first type of exercise isdisplayed without the progress information. In some embodiments, thesecond view of the three-dimensional environment evolves throughout theperformance of the first type of exercise by the first user. In someembodiments, the progress information is continuously updated throughoutthe performance of the first type of exercise by the first user (e.g.,overlaying the changing second view of the three-dimensionalenvironment). In some embodiments, the progress information is displayedin response to detecting that values of one or more progress parametershave met preset threshold values (e.g., a target distance is achieve, athreshold score is reached, an exercise routine is completed, etc.).

Visually presenting progress information of the first type of exercisethat is performed by the first user when the second view of thethree-dimensional environment is displayed provides improved visualfeedback to the user (e.g., improved visual feedback related to progressinformation of the first type of exercise). Providing improved feedbackenhances the operability of the device, which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, in accordance with a determination that the firstuser is facing a first direction in the physical environment, thecomputer system displays a first subset of the first set of virtualcontent corresponding to the first type of exercise without displaying asecond subset of the first set of virtual content, and in accordancewith a determination that the first user is facing a second direction inthe physical environment different from the first direction (e.g.,opposite from the first direction, at a non-zero angle from the firstdirection, etc.), the computer system displays the second subset of thefirst set of virtual content (e.g., virtual open water 7406 not shown inview 7408 in FIG. 7O, virtual mountain trail 7412 not shown in view 7410in FIG. 7P, etc.) corresponding to the first type of exercise withoutdisplaying the second subset of the first set of virtual. For example,in some embodiments, the second view of the three-dimensionalenvironment is an immersive view with virtual objects in directions allaround the first user (e.g., spanning an angle that is wider than theuser's field of view, so when the user turns his/her head around, he/shesees different portions of the virtual environment). In someembodiments, the computer system outputs sound effects using animmersive audio output mode, such as a surround sound mode, or spatialaudio mode that provide localized sound at positions corresponding tosound producing virtual objects (e.g., cheering crowds, ocean waves, avirtual coach, etc.) in the virtual environment.

Displaying a first subset of the first set of virtual contentcorresponding to the first type of exercise without displaying a secondsubset of the first set of virtual content in accordance with adetermination that the first user is facing a first direction in thephysical environment, and displaying the second subset of the first setof virtual content corresponding to the first type of exercise withoutdisplaying the second subset of the first set of virtual in accordancewith a determination that the first user is a second direction in thephysical environment different from the first direction, displays anappropriate subset of the first set of virtual content corresponding tothe first type of exercise when a set of conditions has been met withoutrequiring further user input (e.g., further user input to navigatethrough the first set of virtual content corresponding to the first typeof exercise). Performing an operation when a set of conditions has beenmet without requiring further user input enhances the operability of thedevice, which, additionally, reduces power usage and improves batterylife of the device by enabling the user to use the device more quicklyand efficiently.

In some embodiments, the first type of exercise is a rowing exercise,the second location is a location with a piece of rowing exerciseequipment (e.g., object 7404, other rowing equipment, etc.) present, andthe second view (e.g., view 7408 in FIG. 7O) of the three-dimensionalenvironment includes a virtual scene with open water (e.g., virtual openwater 7406 in FIG. 7O). In some embodiments, the first criteria furtherincludes a requirement that the first user sits down in the rowingexercise equipment and puts his/her hands on the ores of the rowingexercise equipment (e.g., as shown in FIG. 7O(A)) in order for the firstcriteria to be met. In some embodiments, the second type of exercise isa walking exercise, and the second location is a location with atreadmill (e.g., object 7402, or other walking equipment, etc.), and thethird view (e.g., view 7410 in FIG. 7P) of the three-dimensionalenvironment includes a virtual scene showing an outdoor walking path(e.g., virtual trail 7412 in FIG. 7P) (e.g., a hiking trail, a lake sidepath, a city street, etc.). In some embodiments, the second criteriafurther includes a requirement that the first user steps onto thetreadmill and takes at least one step. Displaying a second view of thethree-dimensional environment that includes a virtual scene with openwater, wherein the first type of exercise is a rowing exercise and thesecond location is a location with a rowing exercise equipment present,provides improved visual feedback to the user (e.g., improved visualfeedback that the first type of exercise is a rowing exercise, improvedvisual feedback that rowing exercise equipment is present at the secondlocation, etc.). Providing improved feedback enhances the operability ofthe device, which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, in response to detecting the movement of the firstuser (e.g., 7002 in FIGS. 7O-7P) from the first location to the secondlocation: in accordance with a determination that the second locationcorresponds to a location associated with a fifth type of exercise and asixth type of exercise: in accordance with a determination that themovement of the first user from the first location to the secondlocation is followed by engagement with a respective type of equipmentassociated with the fifth type of exercise by the first user at thesecond location, the computer system displays a sixth view of thethree-dimensional environment, wherein the sixth view of thethree-dimensional environment includes a fifth set of virtual contentcorresponding to the fifth type of exercise (e.g., volleyball, tennis,elliptical machine, etc.), wherein the fifth set of virtual content isdifferent from the first set of virtual content and the second set ofvirtual content, and wherein the fifth set of virtual content replacesat least a portion of a fifth representation of a fifth portion of thephysical environment. In some embodiments, in response to detecting themovement of the first user from the first location to the secondlocation: in accordance with a determination that the movement of thefirst user from the first location to the second location is followed byengagement with a respective type of equipment associated with the sixthtype of exercise by the first user at the second location, the computersystem displays a seventh view of the three-dimensional environment,wherein the seventh view of the three-dimensional environment includes asixth set of virtual content corresponding to the sixth type of exercise(e.g., basketball, fencing, exercise bike, etc.), wherein the sixth setof virtual content is different from the first set of virtual content,the second set of virtual content, and the fifth set of virtual content,and wherein the sixth set of virtual content replaces at least a portionof the fifth representation of the fifth portion of the physicalenvironment (e.g., the fifth portion of the physical environment isassociated with both the fifth type of exercise and the sixth type ofexercise). This is illustrated in FIGS. 7O and 7P, where when the user7002 moves from one location that includes the object 7404 thatcorresponds to a first type of exercise (e.g., rowing, boating, etc.) toanother location that includes the object 7402 that corresponds to asecond type of exercise (e.g., hiking, walking, etc.), the virtualcontent (e.g., virtual open water 7406, or other virtual content, etc.)in the view 7408 is replaced with the virtual content (e.g., hikingtrail 7412, or other virtual content, etc.) in the view 7410, inaccordance with some embodiments.

Displaying a sixth view of the three-dimensional environment, whereinthe sixth view of the three-dimensional environment includes a fifth setof virtual content corresponding to the fifth type of exercise, inaccordance with a determination that the movement of the first user fromthe first location to the second location is followed by engagement witha respective type of equipment associated with the fifth type ofexercise by the first user at the second location, and displaying aseventh view of the three-dimensional environment, wherein the seventhview of the three-dimensional environment includes a sixth set ofvirtual content corresponding to the sixth type of exercise, inaccordance with a determination that the movement of the first user fromthe first location to the second location is followed by engagement witha respective type of equipment associated with the sixth type ofexercise by the first user at the second location, displays anappropriate view of the three-dimensional environment with a respectiveset of virtual content corresponding to the respective type of exercisewhen a set of conditions has been met without requiring further userinput (e.g., further user input to select or navigate between views ofthe three-dimensional environment and/or sets of virtual contentcorresponding to respective types of exercises). Performing an operationwhen a set of conditions has been met without requiring further userinput enhances the operability of the device, which, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

In some embodiments, the second view (e.g., view 7408, view 7410, etc.in FIGS. 7O-7P) of the three-dimensional environment includes a virtualrepresentation of the first user (e.g., user 7002 in FIGS. 7N-7P, oranother user, etc.) that is shown to perform the first type of exercise(e.g., based on previous best records of the first user, based on apreset configuration of the first user for the first type of exercise,etc.) in competition with the first user. In some embodiments, the thirdview of the three-dimensional environment includes a virtualrepresentation of the first user that is shown to perform the secondtype of exercise (e.g., based on previous best records of the firstuser, based on a preset configuration of the first user for the secondtype of exercise, etc.) in competition with the first user. Displayingthe second view of the three-dimensional environment, including avirtual representation of the first user that is shown to perform thefirst type of exercise in competition with the first user, providesimproved visual feedback to the user (e.g., improved visual feedbackregarding the first type of exercise, improved visual feedback regardingcharacteristics of the first user's performance of the first type ofexercise, etc.). Providing improved feedback enhances the operability ofthe device, which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, the second view (e.g., view 7408, view 7410 inFIGS. 7O-7P, etc.) of the three-dimensional environment includes avirtual representation of at least a second user different from thefirst user (e.g., user 7002 in FIGS. 7N-7P, or another user, etc.) thatis shown to perform the first type of exercise in competition with thefirst user. In some embodiments, the third view of the three-dimensionalenvironment includes a virtual representation of at least a second userdifferent from the first user that is shown to perform the second typeof exercise in competition with the first user. Displaying the secondview of the three-dimensional environment, including a virtualrepresentation of at least a second user different from the first userthat is shown to perform the first type of exercise in competition withthe first user, provides improved visual feedback to the user (e.g.,improved visual feedback that at least the second user is alsoperforming the first type of exercise, improved visual feedbackregarding the user's performance of the first type of exercise relativeto the second user's performance of the first type of exercise, etc.).Providing improved feedback enhances the operability of the device,which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

It should be understood that the particular order in which theoperations in FIG. 12 have been described is merely an example and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 8000, 9000, 10000, and 11000) are also applicable in ananalogous manner to method 12000 described above with respect to FIG. 12. For example, the gestures, gaze inputs, physical objects, userinterface objects, controls, movements, criteria, three-dimensionalenvironment, display generation component, surface, representation ofphysical object, virtual objects, and/or animations described above withreference to method 12000 optionally have one or more of thecharacteristics of the gestures, gaze inputs, physical objects, userinterface objects, controls, movements, criteria, three-dimensionalenvironment, display generation component, surface, representation ofphysical object, virtual objects, and/or animations described hereinwith reference to other methods described herein (e.g., methods 8000,9000, 10000, and 11000). For brevity, these details are not repeatedhere.

The operations described above with reference to FIGS. 8, 9A-9B, 10, 11, and 12 are, optionally, implemented by components depicted in FIGS.1-6 . In some embodiments, aspects/operations of methods 8000, 9000,10000, 11000, and 12000 may be interchanged, substituted, and/or addedbetween these methods. For brevity, these details are not repeated here.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best use the invention and variousdescribed embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method, comprising: at a computer system thatis in communication with a first display generation component and one ormore first input devices: displaying a first computer-generatedexperience that includes displaying one or more virtual elements in athree-dimensional environment with a first appearance; while displayingthe one or more virtual elements in the three-dimensional environmentwith the first appearance, receiving biometric data corresponding torespiration of a first user; and in response to receiving the biometricdata corresponding to the respiration of the first user: in accordancewith a determination that the biometric data corresponding to therespiration of the first user meets first criteria, displaying the oneor more virtual elements in the three-dimensional environment with asecond appearance that is different from the first appearance; and inaccordance with a determination that the biometric data corresponding tothe first user does not meet the first criteria, continuing to displaythe one or more virtual elements in the three-dimensional environmentwith the first appearance.
 2. The method of claim 1, including: whiledisplaying the one or more virtual elements in the three-dimensionalenvironment with the second appearance, receiving first updatedbiometric data corresponding to updated respiration of the first user;and in response to receiving the first updated biometric datacorresponding to the updated respiration of the first user: inaccordance with a determination that the first updated biometric datacorresponding to the updated respiration of the first user meets secondcriteria different from the first criteria, displaying the one or morevirtual elements with a third appearance that is different from thesecond appearance; and in accordance with a determination that the firstupdated biometric data corresponding to the updated respiration of thefirst user meets the first criteria and does not meet the secondcriteria, continuing to display the one or more virtual elements withthe second appearance.
 3. The method of claim 1, including: whiledisplaying the one or more virtual elements with a respective appearanceof at least the second appearance and a third appearance different fromthe second appearance, receiving updated biometric data corresponding toupdated respiration of the first user, wherein the respective appearanceis different from the first appearance; and in response to receiving theupdated biometric data corresponding to updated respiration of the firstuser: in accordance with a determination that the updated biometric datacorresponding to the updated respiration of the first user does not meetrespective criteria that were met to transition into displaying the oneor more virtual elements in the three-dimensional environment with therespective appearance, displaying the one or more virtual elements witha respective appearance of the one or more virtual element that is usedprior to displaying the one or more virtual elements in thethree-dimensional environment with the respective appearance.
 4. Themethod of claim 1, wherein the biometric data includes a respirationrate of the first user and the first criteria include a criterion thatis met when the respiration rate of the first user is below a firstthreshold respiration rate in order for the first criteria to be met. 5.The method of claim 1, wherein the first criteria include a requirementthat the biometric data satisfy one or more preset threshold values forat least a threshold amount of time in order for the first criteria tobe met.
 6. The method of claim 1, wherein: displaying the one or morevirtual elements in the three-dimensional environment with the firstappearance includes displaying virtual content at respective firstpositions that correspond to locations of one or more first portions ofa physical environment, while maintaining display of a representation ofone or more second portions of the physical environment; and displayingthe one or more virtual element in the three-dimensional environmentwith the second appearance includes displaying virtual content at therespective first positions that correspond to the locations of the oneor more first portions of the physical environment and at respectivesecond positions that correspond to at least some of the one or moresecond portions of the physical environment.
 7. The method of claim 1,including: in response to receiving the biometric data corresponding tothe respiration of the first user and in accordance with a determinationthat a change in the biometric data corresponding to the respiration ofthe first user is progressing toward meeting the first criteria:gradually reducing visual emphasis of at least a portion of arepresentation of a physical environment that had been visible via thefirst display generation component while the one or more virtualelements are displayed with the first appearance, wherein displaying theone or more virtual elements in the three-dimensional environment withthe second appearance includes displaying virtual content of the firstcomputer-generated experience at a position corresponding to the portionof the representation of the physical environment such that the portionof the representation of the physical environment ceases to be visiblevia the first display generation component.
 8. The method of claim 1,including: in response to receiving the biometric data corresponding tothe respiration of the first user and in accordance with a determinationthat a change in the biometric data corresponding to the respiration ofthe first user is progressing toward meeting the first criteria:changing a visual property of at least a portion of a representation ofa physical environment that had been visible via the first displaygeneration component while the one or more virtual elements aredisplayed with the first appearance in the three-dimensional environmentby an amount that corresponds to the change in the biometric datacorresponding to the respiration of the first user.
 9. The method ofclaim 1, including: in response to receiving the biometric datacorresponding to the respiration of the first user and in accordancewith a determination that a change in the biometric data correspondingto the respiration of the first user is progressing toward meeting thefirst criteria: expanding display of virtual content onto at least aportion of a representation of a physical environment that had beenvisible via the first display generation component while the one or morevirtual elements were displayed with the first appearance, by an amountthat corresponds to the change in the biometric data corresponding tothe respiration of the first user.
 10. The method of claim 1, whereinthe first criteria include a criterion that the first user makes lessthan a threshold amount of movement of a first type when the biometricdata is being received in order for the first criteria to be met. 11.The method of claim 1, including: while displaying the firstcomputer-generated experience including displaying the one or morevirtual elements with the second appearance, detecting movement of afirst type being performed by the first user; and in response todetecting the movement of the first type being performed by the firstuser: in accordance with a determination that the movement of the firsttype exceeds a preset threshold amount of movement, redisplaying the oneor more virtual elements with the first appearance.
 12. The method ofclaim 1 including: while displaying the one or more virtual elementswith the second appearance, detecting movement of a first type beingperformed by the first user; and in response to detecting the movementof the first type being performed by the first user: in accordance witha determination that the movement of the first type exceeds a presetthreshold amount of movement, switching from displaying the one or morevirtual elements with the second appearance with a first viewpoint todisplaying the one or more virtual elements with the second appearancewith a second viewpoint different from the first viewpoint.
 13. Themethod of claim 1, wherein a transition from displaying the one or morevirtual elements in the three-dimensional environment with the firstappearance to displaying the one or more virtual elements in thethree-dimensional environment with the second appearance is a discretetransition that is made at a point in time that corresponds to a timethat the first criteria are met.
 14. The method of claim 1, wherein thefirst computer-generated experience displayed with the one or morevirtual elements having the first appearance depicts a first virtualenvironment and the first computer-generated experience displayed withthe one or more virtual elements having the second appearance depicts asecond virtual environment that has more virtual depth than the firstvirtual environment.
 15. The method of claim 1, wherein: displaying theone or more virtual elements in the three-dimensional environment withthe first appearance includes displaying the first computer-generatedexperience with at least a first visual characteristic that changes inaccordance with a change in the biometric data received while displayingthe one or more virtual elements with the first appearance; anddisplaying the one or more virtual elements in the three-dimensionalenvironment with the second appearance includes displaying the firstcomputer-generated experience with at least a second visualcharacteristic that changes in accordance with a change in the biometricdata received while displaying the one or more virtual elements with thesecond appearance.
 16. The method of claim 1, including: in response toreceiving the biometric data corresponding to the respiration of thefirst user: in accordance with a determination that the biometric datacorresponding to the respiration of the first user meets the firstcriteria, changing an audio output mode from a first audio output modeto a second audio output mode, wherein the first audio output mode hasfewer computationally-controlled variables than the second audio outputmode.
 17. A computer system, comprising: a first display generationcomponent; one or more input devices; one or more processors; and memorystoring one or more programs, wherein the one or more programs areconfigured to be executed by the one or more processors, the one or moreprograms including instructions for: displaying a firstcomputer-generated experience that includes displaying one or morevirtual elements in a three-dimensional environment with a firstappearance; while displaying the one or more virtual elements in thethree-dimensional environment with the first appearance, receivingbiometric data corresponding to respiration of a first user; and inresponse to receiving the biometric data corresponding to therespiration of the first user: in accordance with a determination thatthe biometric data corresponding to the respiration of the first usermeets first criteria, displaying the one or more virtual elements in thethree-dimensional environment with a second appearance that is differentfrom the first appearance; and in accordance with a determination thatthe biometric data corresponding to the respiration of the first userdoes not meet the first criteria, continuing to display the one or morevirtual elements with the first appearance.
 18. A computer readablestorage medium storing one or more programs, the one or more programscomprising instructions that, when executed by a computer system thatincludes a first display generation component and one or more inputdevices, cause the computer system to perform operations, including:displaying a first computer-generated experience that includesdisplaying one or more virtual elements in a three-dimensionalenvironment with a first appearance; while displaying the one or morevirtual elements in the three-dimensional environment with the firstappearance, receiving biometric data corresponding to respiration of afirst user; and in response to receiving the biometric datacorresponding to the respiration of the first user: in accordance with adetermination that the biometric data corresponding to the respirationof the first user meets first criteria, displaying the one or morevirtual elements in the three-dimensional environment with a secondappearance that is different from the first appearance; and inaccordance with a determination that the biometric data corresponding tothe respiration of the first user does not meet the first criteria,continuing to display the one or more virtual elements with the firstappearance.